I have always wondered the same thing. I came to the conclusion that it's pretty difficult to determine that.
I lived and worked as a software developer through the Y2K "crisis" (although I wasn't working on solving the crisis myself). Everyone was very worried about it. Nothing really went wrong in the end.
Was that because there was no problem? Or because everyone was worried about it and actually solved the problem? I don't think it's easy to tell the difference really.
>> It's only hard to tell if you don't talk to the developers who were working in the late 90s.
I think you mean "working ON it". Talking to developers as a broad group from that time wouldn't necessarily produce any useful information.
The person you replied to was himself a developer working in the late 90s. During the late 90s, I talked to a lot of developers, but only a small percentage of them were on Y2K jobs.
There were a lot of Y2K related tasks in the course of many developers generally activity. Where I worked at the time, there were not developers solely dedicated to Y2K work. The Y2K issue pretty much caused an employment boom for software developers though.
I was fixing Y2K bugs at a company founded in 1997. Unless you never touched date stuff, I can't imagine being oblivious to Y2K issues working in IT in 1999.
The startups I worked in around that time - we were using recent hardware with 4 digit years and unconcerned about Y2K issues.
So while we were -aware- of the Y2K issue, it didn't impact any of us in a concrete fashion. We would talk about people we knew on Y2K projects, which were mostly mission critical legacy systems.
So it's not inconceivable for devs in the 90s to have only cursory awareness of the -real- issues that the people who worked on Y2K projects were facing and solved.
We were all on Y2K jobs, in the sense that for the decade of the 90s you'd get looked at very weirdly if you tried to produce code and systems using two digit years.
I guess I have a different perception. If you were trying to produce code in the 90s using two digit years, the weird looks had less to do with Y2K than "why would you save incomplete data to save a couple of bytes"?
For context, most of the devs I worked with at that time finished their CS degrees within the previous 10 years, and were not legacy systems programmers. Doing 2 digit years was inconceivable to them.
I think so too, but there is some evidence of a lot of legacy systems needing critical updates. Manufacturing, oil processing, military hardware and other stuff. The problem was that it was nearly impossible to understand how failures would manifest. Just that there will be failures and some of them might be very critical. Luckily, all the important stuff got fixed in time that everything went pretty smoothly, but there were still some problems in non-critical systems.
That said, if we need to jump start the economy again, maybe we could come up with a fake problem this time. How about we say that 2025 will be a huge problem for many critical IT systems? Or does a timestamp does produce a nice round number in the near future that is a good sell?
No. It was not. I was a software developer in a large US Bank at the time. We had already dealt with it years ago for critical systems. All the banks had.
Right, everyone was well aware. But I think there was a bit of "We don't need to worry because we're replacing the mainframe with the new ERP system in 1997", and then whoops the mainframe was still running.
I was as well. Agree that the core systems were fine and extremely thoroughly tested, but all of the supporting applications/infrastructure were questionable. I had quit my job a couple months prior to go into contracting. They were one of my first customers, and the contract was contingent on me remaining on full time until after the new year weekend. Win win.
I had the same experience working in a large investment bank. There were a lot of very expensive management consultants involved who strung out everything for the sake of their hourly rate. I wasted months proving to their satisfaction that my recently written and robust code was OK.
It was a VERY widely held belief by the general public that Y2K only applied to real time clocks and wall time.
However arguably most dates in corporate IT work are involved in some level of forecasting and prediction and future planning.
In reality, starting in 1970 anyone writing an amortization table program for a 30 year mortgage had to work around Y2K. Anyone dealing in any way with the expiration date for a twenty year term life insurance policy had to start caring about Y2K in 1980. Even a mere net-30 business to business payment account either broke or not in nov-1999. Even on Dec 31 1999 it was hilariously charming how people all around the world thought all computers were located in THEIR timezone and thus any real time clock type failures would occur at precisely midnight local time where they live as opposed to where the computer is actually located. Due to the miracle of UTC time anything bad would have happened to our stuff early in the day while I was eating a late dinner, not when the operations center overstaffed during local timezone 3rd shift.
I was working at a telco at the time and we were very worried and overstaffed over Y2K, our stuff was fine, but we were pretty worried about rioters and such if anyone ELSE failed, like maybe the power co. Hilariously the power co people were probably overstaffed over Y2K, despite knowing their stuff was fine, they were likely worried about those telco goofballs failing thus losing SCADA links to their substations, LOL.
In the end it seems pretty much nothing failed anywhere, as I recall. Or the failure rate for that day, was no higher than any other average calendar date.
From someone who went through it and dealt with code, it was a real problem but I also think it was handled poorly publicly. The issues were known for a long time, but the media hyped it into a frenzy because a few higher profile companies and a lot of government systems had not been updated. In fact, there were still a number of government systems that were monkey patched with date workarounds and not properly fixed well into the 2000's (I don't know about now but it wouldn't shock me).
There was a decent influx of older devs using the media hype as a way to get nice consulting dollars, nothing wrong with that, but in the end the problem and associated fix was not really a major technical hurdle, except for a few cases. It is also important to understand a lot of systems were not in a SQL databases at the time, many were in ISAM, Pic, dBase (ouch), dbm's (essentially NoSql before NoSql hype) or custom db formats (like flat files etc) that required entire databases to be rewritten, or migrated to new solutions.
My 2 cents, it was a real situation that if ignored could have been a major economic crisis, most companies were addressing it in various ways in plenty of time but the media latched on to a set of high profile companies/government systems that were untouched and hyped it. If you knew any Cobol or could work a Vax or IBM mainframe you could bank some decent money. I was mainly doing new dev work but I did get involved in fixing a number of older code bases, mainly on systems in non-popular languages or on different hardware/OS because I have a knack for that and had experience on most server/mainframe architectures you could name at that time.
It a shame that we as engineers can't just say "this maintenance is required to fix this known issue. It's not a huge deal but will cause trouble if it's not dealt with". Instead we have to be all doom and gloom and tell management that the company/world will end.
Colleague was just dealing with a clients of their who was still using TLS1.0. They're running classic ASP on Window Server 2008, and can't (effectively) migrate. Colleague had been raising the alarm for months (since they started on the project) that "this is going to mean all your systems will stop working in early 2020" but no one seemed to care or understand.
They did, last week, put in ... haproxy as an SSL terminator in front of the main server, and will test a switch over this week. This was 8 months of foot-dragging for about 3 hours of setup/config, and a couple more hours of testing. When all your clients are hitting a web server, and their browsers will all stop rejecting your certs, things will get ugly fast - as in "your business will effectively stop functioning". It just sounded like "doom and gloom" but... how do you message this effectively? It requires the receiving parties to actually understand the impact of what you're saying, regardless of terms you use.
I hope your colleague had a paper trail of the alarms he raised when it came time to point fingers.
For executives with limited IT experience I honestly don't know if there is a good solution, other than having them deal with the disaster and having a clear paper trail that points the finger in their direction. They won't make the same mistake twice.
Money, Money is the way you talk that makes business listen.
You say "If we don't solve this by X date, we are looking at losing the ability to take in revenue and possible law suits for failure to perform. It will take Y amount of time to accomplish this"
If you don't couch things in terms of time, money, resources, client impact. They will not care. You can say "Hey, it is super bad that we are running Tomcat 7.0.0 there are a lot of security vulnerabilities" and what they will hear is "blah, blah, blah, we can delay this".
We've been forced to deal with the TLS issue by our software product's customers. Some of our technically-minded folks have been raising the issue for a while but new features are sexy and sell, and fixing not-yet-broken code doesn't. Amazing/discouraging that only the threat of immediate loss of six figures of income gets any attention.
Assuming the entire org is using a captive proxy with an installed CA, identify/isolate every upper-level management person's system(s) in some way, and reroute their access to all internal applications through a deliberately horribly misconfigured HTTPS reverse proxy running TLS 1.0. Their modern browser will explode.
"AAAA, sorry, that thing we've been telling you about for months hit us earlier than we thought it would but thankfully it's only hit all the internal stuff"
"Wait so this is what all of our customers will see?"
"Yes, all of them. Nobody will be to reach us, none of our APIs will work, and we will also break our SLAs on every single contract."
---
You probably want to use GPO to disable bypassing the security warning prompt, and/or set up HSTS for your domains beforehand.
There's less cause to use small data sizes for timestamps and date codes now. Storage has grown by orders of magnitude, the idea that a numeric data type would only be large enough to store a 2-digit year or that you would want to save disk space by abbreviating an extra 2 letters is foreign to a lot of new developers. And the 20-year-old systems are slowly dissapearing...
More importantly we have figured out algorithms for time that account for timezone, daylight savings time, different calendar systems, and probably something else I'm not aware of. These all work by counting seconds since a fixed date.
Note that the algorithms were known since at least the late 1960s. Storage space even then shouldn't have been enough of a concern. However people still screw it up all the time.
This is the accurate point. We had epoch date tracking since the 1960s or so and the coding of a two digit year was out of laziness and convenience in most cases. And not bashing anyone, but most of the systems I saw using the lazy method were more business systems where there wasn't thoughtful engineering a lot of the time, but more just get it done. Those two don't have to be mutually exclusive IMO, but in the 70s/80's it seems a lot of the time it was, simply because IMO we had predominately really two classes of developers, scientific and business where business devs took less of an engineering mindset and more get it done in a specific timeframe so they "hacked" it out. Plus it was new, so stuff happens.
Sadly this stuff still goes on today, just in slightly different ways. Data type and data structure choices are critical, and it isn't as important to me that someone know how to write an algorithm for a RB-tree (or pick your DS), but it is critical an engineer knows which data structure/type to choose for specific use cases. This is one of my biggest complaints with the way a lot of engineering interviews are structured today, they want you to regurgitate known algorithms which is a waste of time and personal memory, find out if the person knows when they'd use a specific data structure and why the rest is easily researched. I mention this because to me the fundamental flaw of the y2k problem was one of data structure and data type choice.
Seconds since the epoch doesn’t work for times in the future. Consider a contract that will terminate at 2025-02-28 12:00 in New York: will that be -05:00 or -04:00? It could change.
Perhaps you could expand on this as I was not involved in Y2K myself. Storing dates with 2 digit years doesn't seem to make much sense as a storage saving mechanism for me.
If you want to store only the year, why not store the years since 1900? That way in a single byte you can go all the way up until 2155. If you want to store it bit-aligned rather than byte-aligned why limit yourself to 00-99? 7 bits can store the year from 1900-2027.
If you want to store day + month + year with a two digit year you have 365.25*100=36525~ options. A two byte value can store up until 65535 options, which can hold all days until 2079.
If you are storing your dates as flat strings (DD-MM-YY or so) then you are already wasting such a large amount of bytes that "storage concerns" seem moot. You are already using 8 bytes to store a date that can be stored in 2 bytes, at that point why not use 10 bytes?
String formatting and display seems more obviously affected to me than storage.
Most input systems were punch cards, which were designed to be fixed-length textual input directly from a user at a card punch machine. Teaching users how to map from “years since 1900” to the correct keys on the card punch is a lot more confusing and error prone than having the computer automatically convert from "71" to 1971.
That makes a lot more sense, but if it is just input then why not do the conversion of the string "71" to the efficient byte representation of 1971 when it is moved into the storage. If the computer can do a mapping of "71" -> 1971 then why can't it also do a mapping of "10-10-71" -> 26214 (days since 1900-01-01).
When 5MB disk storage system is the size of a small car and costs $50k, you fight for bytes. Or if your data is stored on magnetic tapes and reading in a dataset can be costed by the kilobyte, takes minutes per megabyte, and requires a million dollar's worth of equipment and dedicated staff, you fight for every byte.
The phone I carry today is hundreds or thousands of times more powerful than the $5,000,000 mainframe I was working around Y2K and the apps on it - most delivered freely - make it far more capable than the raw processing power difference would indicate.
If you are fighting for bytes then storing dates as "DD-MM-YY" strings is very inefficient, as you are wasting a factor 4X on storage compared to just storing the days since 1900 as a 2-byte integer. I understand that storage used to be expensive, but I don't understand where the two-digit year comes into play as I can't envision an efficient storage mechanism that is limited to exactly the years 1900-1999.
You need also to have the code to do the conversion. If you store, in a byte, the years since 1900 (or 1970) then every time you load and/or process a date to be displayed you need to add 1900 (or 1970) and then convert from int to str which takes time (machines were slow) and code.
And the code takes space.
You may think the space is trivial, but I remember working for nearly two weeks on a commercial system to save around 15 bytes on a system, and it was worth it.
Then the culture was embedded ... you worked to save bytes, and didn't do things that would cost more bytes. And the systems worked, and didn't need updated, so they persisted.
One of my tasks for Y2K prep was to modify an in-house-developed record-based "database" to accommodate a the century change. Originally developed in the '60s the year had originally been a single digit because everyone thought commercial DBMS' would be viable in few years so there would be a replacement before the decade rolled over. :-| They'd grafted decade handling into the thing before I got there so it could survive the '70s because they had a lot of expensive data in various file sets that contained the data. Converting to a commercial database was expensive and represented a huge business risk, so...
If you have eight bit bytes, you would of course store these as three two digit BCD numbers requiring one byte each, and need no conversion logic. (The code for printing BCD numbers is already there, because you need it to print
amounts of money anyway).
This is a good question. I found a copy of Fujitsu COBOL 85 manual [1] (the next release was post y2k, in 2002) and was surprised to see that CURRENT-DATE, INTEGER-OF-DATE use 4-digit years.
However some functions from that time (eg ACCEPT, p. 544) do use 2-digit years. I think you have it exactly right, it was about facilitating string formatting/display: ACCEPT took input from the terminal screen. (I think it is, approximately, gets() + strptime(), though I don't see any talk of error handling). Programmers probably just stuffed the result into a record before writing to DASD (disk), optimizing for programmer time and program complexity over storage.
Perhaps 2-digit years for ACCEPT were an optimization for data entry workers?
A modern z/OS COBOL guide [2] has 4-digit years throughout as far as I can see.
Whilst I'm here, I'm also going to say: "VSAM", "PDS", TSO", "LRECL" and "ABEND" (just because I haven't heard those words for a quarter century and now I'm feeling nostalgic).
> If you want to store only the year, why not store the years since 1900? That way in a single byte you can go all the way up until 2155.
You're making an assumption in that statement that there's a byte and it's eight bits in length. This was not always true. Punch cards didn't really work that way, and fair number of CPU's didn't either.
Early machines used various decimal systems and 6-bit characters were widespread also. This explains a bit of why the C language and descendants often have native support for Octal literals and Unix permissions are built around three bit groupings.
On a personal note, even in the very early 90's, I also remember using a CDC machine (a descendant of Seymour Cray's 6600) that supported 60-bit words with 6-bit characters. Pressure to move to 64 bit words with 8-bit bytes resulted in dual mode design that could be booted either way.
CDC...if it was like the CDC system I worked on back in the early 80s, the 6-bit chars (64 distinct chars, including control chars) did not include lower case Latin. Our system was munged so that lower case alphabetic chars could be represented by prefixing a backslash. I wrote my dissertation on that system, and fortunately the editor I used displayed a backslashed char as lower case.
>> And the 20-year-old systems are slowly dissapearing
There's still COBOL code out there running core business systems that's older than I am, and I'm closer to 60 than I am to 50.
There was a related story here on HN recently. A system was assuming a 101-year-old was actually only 1 year old, presumably because of 2 digit arithmetic.
One system I worked on showed the date as year 3900, because the function for getting current year would return 2 digit for year < 2000, and full 4 digit for >= 2000. So the code did 1900 + GetYear(). Replaced it with just calling GetFullYear() or something like that.
At the time I was managing a dBase / FoxPro medical software package...we were a small staff who had to come up with Y2K mitigation on our own.
Our problem is we only had source code for "our" part of the chain...other data was being fed into the system from external systems where we had no vendor support.
Thus our only conceivable plan was to do the old:
If $year<10;
date="20$year"
else
date="19$year"
It worked in 99.9% of the cases which was enough for us to limp thru and just fix the bad cases by hand as they happened. Eventually we migrated off the whole stack over the next few years so stopped being a problem. I'm sure many mitigation strategies did the same....
This is what I remember a lot of too. While they are little hacks which are imperfect, they bought companies enough time to resolve the issue more thoroughly.
I still love the fact that if you only implemented the first rule or had the knowledge to implement all 3 rules, it would totally work, but if you implemented 2 of the 3 rules you were wrong.
It taught me a great lesson about results not proving correctness as how you got there could bite you later.
I don't get why people don't know all three rules. In elementary school when the calendar is taught, the complete rules of the leap year were simply taught by the teacher. We even joked about people born on the 29th of February. Why wasn't this taught everywhere?
Heh, this sentence seems particularly relevant to this whole discussion...
> This method works fine for the year 2000 (because it is a leap year), and will not become a problem until 2100, when older legacy programs will likely have long since been replaced.
Man, I've just remember that I had to fix this kind of bug in 2010 and I am sure I just bumped the number to 20. I guess somebody had to be fixing it lately, I just hope they didn't just replace 2 with 3.
It's been a very long time since I've worked with dBASE/FoxPro, but from what I remember it stores data in a fixed-width format. So for a column to store an integer, it will zero pad the front of it to fit the width of the column.
I'd argue if it wasn't for the media "doom and gloom" a lot of companies (maybe even the vast majority), would not have had their C level folks divert money towards fixing the issue.
The simple reason is that any one company may have been liable for a very small portion, which if it failed, would not have caused much trouble. But that failure combined with many other failures down the entire chain of connected and unconnected software would have added up to something much greater than the sum of parts.
We saw similar stuff happen with Flash, Silverlight (which wasn't as reported a concern since silverlight usage was so low, but I saw it within my company).
The media pressure was a significant reason why every company needed to have a plan to deal with it.
I was working at a telecommunications startup in 1999. They were founded in 1997. A big part of what I was doing was fixing Y2K bugs.
That said, none of the bugs would have been critical to the operations of the services. Everything was in the billing systems and I think if unfixed it would have been more of a reputation hit than anything.
The descriptivists don't even win, they just accept reality. Language is not constructed by individual humans or small groups, but socially over entire societies. Fighting this makes prescriptivists look like grumpy armchair academics and still doesn't stop me from saying "le weekend".
You should stop reading this discussion here, for your own peace of mind. Because there's someone writing quotation marks and then immediately writing "quote-unquote" only a few bullet points ahead. (-:
"Begging the question" comes from a centuries-old mistranslation which somehow became a shibboleth. I could probably come up with a dumber way for a phrase to become fixed as "proper usage" but it would take me a while.
Just imagine if no one had ever lifted a finger to fix any of the bugs. We only talk about it being a scam because everyone collectively did such a great job mitigating it in time.
There is a certain class of work in the IT/software world that is utterly thankless - nothing goes wrong and people wonder what they pay you for, something goes wrong and the very same questions get asked.
I don't think that's true. I've never heard that about any other industry. Certainly not about janitorial work. Doctors curing a patient, lawyers winning a big case, secretaries keeping the schedule up to date, teachers with kids who pass their tests, truck drivers who deliver on time. all of that is plainly visible. IT is a field in which few people see the results or understand the effort required.
it's true of any industry or organizational structure that is considered a cost center.
As for your examples, those are all boolean - they either do it successfully or they don't, and if they do, it's what they are paid for, and if they don't, they catch hell. There is nothing intrinsic about IT that differs.
Yes. It's like if you're on a ship and you see an iceberg in the distance. If you shout iceberg, and change the direction just a little, no crisis. Without that small change, big problem.
People were talking about Y2K years ahead of time. Lots of changes to code were made. A few little bugs slipped through, but not many, and everyone knew how to fix them. No crisis. Without the many code changes, big problem.
This is a tricky situation indeed, and I suspect that many governments face the same conundrum right now with the pandemic. If you take the right measures early on and everything goes well many people in the opposition will say that you just did it for the media circus in order to get attention, wasting people's time and tax money. If you don't take action and things get out of hand then you're blamed for not taking action earlier. Damned if you do, damned if you don't.
In the end you end up in this perverse situation where you have to wait long enough for the public to understand what's at stake but not long enough that you won't be able to keep things under control. Quite the tightrope trick.
No it was not a crisis. There were plenty of bugs in legacy systems that needed to be fixed, but legacy systems have bugs all the time, for example when the dates for daylight savings time changed. The general public was not well informed and also generally didn't have a software background to understand the problem.
I've also found myself musing on a similar question, but one where you may have a different temporal perspective at this particular moment: In six months, are we going to collectively believe that the Coronavirus was nothing and we massively overreacted to it? Because if we do react strongly, and it does largely contain the virus, that will also be "proof" (quote-unquote) that it wasn't anything we needed to be so proactive about in the first place.
Unsurprisingly, humans are not good at accounting for black swan events, and even less so for averted ones.
In my opinion (emphasis opinion), "black swan" includes the concept of timing... that a pandemic would occur is inevitable, but you have no idea on the timing. Market crashes are inevitable, but you have no idea on the timing. Volcanic eruptions are inevitable, but you have no idea on the timing. etc.
Things that are inevitable only when you encompass time spans longer than a human life (it has been approximately one and a half average human lifespans since the previous pandemic) may be predictable at that large aggregate scale, but on useful scales they are not. Or, to put it another way, if you've been shorting the market since 1918 for the next pandemic crash, you went bankrupt a long time ago.
Y2K is only a black swan for those not in the industry, since that one is obviously intrinsically timing based. The UNIX timestamp equivalent is also equally predictable to you and I, but to the rest of the world will seem even more arbitrary if it's still a problem by then. (At least Y2K was visibly obviously special on the normal human calendar.) But I wouldn't claim the term for that; call it a bit of sloppiness in my writing.
> Because if we do react strongly, and it does largely contain the virus, that will also be "proof" (quote-unquote) that it wasn't anything we needed to be so proactive about in the first place.
Even if every other country has Y2K levels of success containing the Coronavirus, we can still point skeptics at the example of places like Italy to prove it was a real threat.
I assume that we'll have a fairly decent idea as to what worked, and what didn't, as the differing responses mean we're effectively carrying out multiple experiments at once.
There were parts of our telecom infrastructure that weren't ready but got fixed before y2k. A certain mobile phone switching vendor (think cell towers, etc.) ran tests a year before to see what happened when it rolled over and the whole mobile network shut down (got in a wedged state where calls would fail, no new calls, signalling died). They fixed it and got customers upgraded in time.
I got my first cell phone in 1996. It worked really well (GSM network) and SMS's were available. So I can easily imagine that if you relied on it, it seemed like a critical system to you personally or as a business.
Imagine that 1% of all software had an issue, across all of the economic tissue of the developed world. Now imagine that this software would start failing, all on January 1st 2000, everywhere around the world. Or better still, not failing, just silently corrupting data.
Just like the crisis we are currently facing in our health systems, it seems unlikely that we would have had enough IT resources to deal with the issues in real-time.
This is one of the cases of a "self-denying-prophecy", much like acid rain. There was an issue, we collectively dealt with it (better yet, we actually anticipated!), and now people are saying that in the end there was no issue.
Yes, the y2k crisis was real, or more accurately, would have been a serious crisis if people had not rushed and spent lots of money to deal with it ahead of time. In many systems it would have been no big deal if unfixed, but there were a huge number of really important systems that would have been a serious problem had they not been fixed. Part of the challenge was that this was an immovable deadline, often if things don't work out you just spend more time and money, but there was no additional time that anyone could give.
The Y2K bug did not become a crisis only because people literally spent tens of billions of dollars in effort to fix it. And in the end, everything kept working, so a lot of people thought it wasn't a crisis at all. Complete nonsense.
Yes, it's true that all software occasionally has bugs. But when all the software fails at the same time, a lot of the backup systems simultaneously fail, and you lose the infrastructure to fix things.
I'm having PTSD just thinking about updates of YY to CCYY in mainframe code. Very real and heavy investment across numerous industries from aviation to financial and more. At a macro level, I remember it being pretty well managed risk and mitigation.
When you start 3+ years before the known threshold date, you're giving yourself the best chances at success. I remember flashing BIOSes for weeks in the computer labs at my university.
EEPROM. And there has always been a reflash method for bricked firmwares for all AMI and Phoenix BIOS' as well. Insert disk with firmware, Hold a key, power on.
Once upon a time we got somewhere in the neighborhood of 1200 dell workstations and not a single one failed the bios upgrade. 10/10 would do again
> people literally spent tens of billions of dollars in effort to fix it
The sceptic inside me is curious as to how much they actually accomplished in their effort to fix it. I mean, yes, they spent billions to get millions of lines of code read, but how many fixes have been made and what would be the cost (when compared to those billions of dollars spent) if they weren't fixed at all.
A good friend of mine was (and still is) a civil/construction engineer. Other than some introduction courses in college, he had never done any programming before. But after losing his job in 1998, he was hired by a major European consulting company to work on the Y2K problem.
With an auditorium full of others who had barely any programming experience, he was given a crash course in Cobol for a month, and then he had to go through thousands of lines of code of bank software that ran on some mainframe and identify all cases where a date was hard-coded as 2 characters.
Extremely tedious work, but there were tons of fixes to be made.
The problem was real, and the billions that were spent on fixing things were necessary.
I’m wondering what happened to the people in meeting rooms who suggested that this could be automated if it needed no deep understanding of the COBOL language or the banking domain..?
EDIT (for clarity): that’s what I would say, and I’m pretty sure many people smarter than I had said it - so I’m just curious what objections made business push forward with the “relatively unskilled” human labour intensive practices instead, because that’s the history we have now.
I'm not a COBOL expert, but I suspect you misunderstand both the difficulty of teaching a machine to understand the problem well enough to automate fixes for it and the available computing resources at the time. This was still the era of Windows 95, the classic Mac, and expensive workstations.
There were tools available, but it was increasing the risk (in a risk-averse atmosphere) to trust them to catch all cases. Try explaining to your superiors why you were given the task of fixing the issue and instead of completely fixing it you saved some money. Probably unacceptable.
Also, much of the remediation could not be automated anyway (for example you may not be able to assume the century is '19' for a given date field which can be in the future. You may need to do a comparison to a cutoff date, or you may only have to enlarge the field knowing that the value is supplied from elsewhere which will itself get fixed), so it fell back to people looking at the code, understanding it just enough, making the required changes, and planning/executing the testing.
I worked for a place that simulated what would have happened with an older device. It would have been pretty catastrophic... easily losses in the billions.
Remember that big old systems were the low hanging fruit in 1970. The business logic was COBOL and assembler, without things like functions. The system may have hundreds of duplicate routines for things like subtracting dates.
Lots of people seem to come up with it now. It's been two decades, and it's brought up by people who use it as an example in their other other agendas:
"look at y2k: we spent so much money, and nothing happened!" (so far, so good) "...therefore, if we hadn't, nothing would have happened anyway!!" (uh, based on...?) "...and it was all a massive scam, just like any massive present issue!!!" (ah, there's the underlying reason)
if you're asking "was the money well spent?" then I'm sure it was as well spent as usual, i.e. mistakes got made, a percentage was wasted and some contractors made very good rates. But in the end we got there.
If you asking "would it be better to have done nothing?" then I would have to say No, of course not. What are you thinking?
>..and what would be the cost (when compared to those billions of dollars spent) if they weren't fixed at all.
I worked on fixing Y2K bugs at the time. It was very methodically and planned, and tested on-site at a space agency. If we hadn't fixed those Y2K issues at the time then weather forecasts would have suffered greatly. The cost of that happening? Hard to calculate I think.
Dear god, reviving the ghosts of IT departments past!!!
Yes, updating JDEdwards AS/400 systems and many a PC was a big project, but i dont recall doing it at that time as being super difficult... frack ist so long ago i cant really recall many of the details other than reporting daily on the number of systems updated.
Also, fuck you google - this was when you were nascent a i converted the entire company to your “minimalist” front page vs yahoo, and a few years later you wasted 3 months of my time interviewing to tell me to expect an offer letter tomorrow and called to then reacind the offer (as i didnt have a degree) and then continuing to contact me for FIVE FUCKING YEARS for the same job that you wouldnt hire me for.
It happened to all of us. But be glad you are not there now.
None of those people contacting you, or phone screening, were Google employees. They were, or worked for, contractors. It didn't matter to them or to Google whether anybody they contacted was hired.
I remember hearing a story on NPR years ago years after Y2K with someone knowledgeable on the subject that addressed this question. He gave basically the same answer you did: a lot of effort went into avoiding disaster and a lot of people treated it after the fact as if it was hype or hysteria. I recall the interview because it changed my significantly thinking on the subject. (I wasn't in the industry at the time of Y2K.)
He also noted that a lot of American firms worked with Indian firms to resolve it. Indian engineers were well suited to the problem, according to him, because they had been exposed to a lot of the legacy systems involved during their studies.
He said an interesting consequence of this was that many American companies concluded they could use lower wage Indian engineers on other software projects so it helped initiate a wave of offshoring in the early 2000s.
I couldn't find the story I heard in the NPR archives, but they do have a number of stories reported right around that time if you want to see how it was being discussed and treated by NPR at the time:
I did find this NPR interview with an Indian official from 2009 that notes Y2K's impact in US-Indian economic relations:
But I think the real turning point in many ways in the U.S. relations was Y2K, the 2000 - year 2000, Indian software engineers and computer engineers suddenly found themselves in the U.S. helping the U.S. in adjusting itself to the Y2K problem. And that opened a new chapter in our relations with a huge increase in services, trade and software and computer industry.
It was also really difficult to detect bugs since no good testing paradigms had been established yet.
This created an exploitable opportunity for the few who knew how to write automated tests.
During the Y2K panic Sun Microsystems (IIRC) announced that they would pay a bounty of ~$1,000 per Y2K bug that anyone found in their software (due to the lack of automated tests, they didn't even know how to find their Y2K bugs)
James Whittaker (a college professor at the time) worked with his students to create a program that would parse Sun's binaries and discover many types of Y2K bugs. They wrote the code, hit run, and waited.
And waited. And waited.
One week later the code printed out it's findings: It found tens of thousands of bugs.
James Whittaker went to Sun Microsystems with his lawyer. They saw the results and then brought in their own lawyers. Eventually there was some settlement.
One of James' students bought a car with his share.
It’s sad to live in a world where people are naive enough to assume that doing the right thing without incentive is commonplace. You must be great to negotiate with.
Yep, this was something that many firms were delaying spending on it. It's pretty frustrating because this seems normal. Only do something about an issue when the time horizon is near rather than take it on when it's further out.
I think a more accurate way of describing the Indian engagement is that a lot of recruiters and Indian firms discovered there was very little vetting of candidates, and that almost anyone could be put forward and start earning money for the recruiter or offshorer.
ahhh i remember as a teenager making sure our board of ed systems were Y2K-compliant. numerous times the principal staff would evacuate the office to take care of some action going on in the halls of the school. leaving me with access to the entire student records db and the VP login and password written on a sticky under the keyboard (not even joking!). in addition this was a laptop so RAS details were conveniently saved in network neighborhood.
this left me at a crossroads. i thought about writing malware that would randomly raise peoples semester grades by 4 points (e.g. C- would became a B+). i thought about mass changing grades. i thought about altering a target group of kids i didn't like. all but the last of the scenarios ended with me getting fingered because I was the smart computer kid. if i didn't touch my grades i would have plausible deniability. i wrote the malware. then i watched office space and decided to think about my actions (and promptly forgot as a horny teenager does). soooo glad i didn't release it because years later i went back into the code and found a leftover debug that would have targeted the only 2 kids in the school who had this letter in their last name.
From what I've heard, a number of IT departments used it as justification to dump legacy systems.
For example, at a former employer of mine, a big justification for getting rid of the IBM-compatible mainframe and a lot of legacy systems which ran on it (various in-house apps written in COBOL and FORTRAN) was Y2K.
In reality, they probably could have updated the mainframe systems to be Y2K-compliant. But, they didn't want to do that. They wanted to dump it all and replace it with an off-the-shelf solution running on Unix and/or Windows. And, for reasons which have absolutely nothing to do with Y2K itself (the expense and limitations of the mainframe platform), it probably was the right call. But, Y2K helped moved it from "wouldn't-it-be-nice-if-we" column into the "must-be-done" column.
Worked in a project to migrate a huge and critical .gov database from an AS400 nobody understood anymore to Microsoft SQL Server. We had to guess some of the values in some bitwise fields through trial and error. Made it with a few weeks to spare - nobody noticed.
Oh, understanding the underlying system, that's comparatively easy. Understanding the bespoke business logic and how it's represented in the system...now that's harder, an order of magnitude at least.
Story time: years ago (2010-ish?), we were doing some light web CMS work for a client. Nothing too complex, except we had to regenerate one section off a data feed that we received in a very peculiar custom format.
Great, everything worked, we finished on time and on budget. And then: "oh, and could you also check this other site? We need some minor tweaks on what your predecessors made". That other site was also consuming that data feed, so we took a peek. It ran in PHP3, walled off from everything, processed its own intermediate languageS, and the output was a massive 200+ page PDF (which was then manually shipped to offset print in large quantities). For Reasons, this had to run daily and had to work on first try. There was no documentation, no comments, and no development environment: apparently this was made directly on prod and carefully left untouched.
Needless to say, the code was massively interdependent, so that the minor tweaks were actually somewhat harder. We did manage to set up a development VM though, and document it a bit - but last I checked, the Monstrous Book Of Monsters Generator still seems to be chugging along in its containment.
COBOL programmers were charging a fortune to be hauled out of retirement to work on this. There was a huge shortage of experienced COBOL devs. And devs who actually understood the specific legacy system involved were even rarer. If the company had customised their system and not kept the documentation up to date or trained new programmers on the system, well, they didn't have a choice. They had to replace it.
To be fair, I've met a lot of young COBOL programmers since starting to work on mainframe systems last year. I think the "elderly COBOL programmer coaxed out of retirement by dumptrucks full of money" dynamic that supposedly dominated the Y2K response is less prevalent these days, and companies that still run mainframe systems just realize they have to hire regular programmers and teach them COBOL and z/OS.
Sorry, didn't mean any slight at you by that. I don't doubt your account, but having no idea how widespread the practice was, its dominance on a large scale is just a supposition on my part.
> From what I've heard, a number of IT departments used it as justification to dump legacy systems.
> In reality, they probably could have updated the mainframe systems to be Y2K-compliant.
Legacy systems weren't always mainframe systems and, in any case a “legacy system” is precisely one you are no longer confident you can safely update for changing business requirements.
A change in a pervasive assumption touching many parts of a system (like “all dates are always in the 20th century”) is precisely the kind of thing that is high risk with a legacy systems.
And the dotcom bust came when it did in part because the Y2K spending was lifting all boats. A bunch of companies shifted from buying too much to buying not much.
I'm not sure why you're being downvoted. It wasn't really related to the dotcom bust per se. But the falloff in Y2K spend was part of the overall drop in IT spend on consultants, system refreshes, etc. We use dot-bomb as a shorthand but there were clearly a number of things that happened in the same general timeframe that made it something of a perfect storm.
>treated it after the fact as if it was hype or hysteria
I agree with everything you and the parent said, but there was also hype and hysteria on top of it all - especially among the communities of preppers, back to the landers, and eco-extremist types. I met entire communities of technophobes who declared I was just ignorant or 'a sheep' and refused to believe me when I tried to explain that the industry was working on the problem and had a solid shot at preventing major catastrophe.
Frightening predictions of power plants going offline for days, erratic traffic lights causing mayhem, food and water supplies being interrupted, even reactors melting down were not uncommon.
As always, there is money to be made by exploiting people's fear. This hysteria was considered an irrelevant side show by most educated people, the way I look at the chem trail people or the way I used to view the anti-vaxxers [1], but the unjustified hysteria was real.
[1] Before I learned their numbers were growing and could threaten herd immunity in some areas.
This. So many parallels with every "looming disaster" since. The media went into frenzy mode a few times, predicting the End Of The World As We Know It and scaring everyone witless.
And a few people did take it really seriously and spend the Millennium in some remote place where they would stand a chance of surviving the collapse of Western Civilisation.
True. The first image that pops into my head when I hear Y2K is a Simpsons joke where airplanes and satellites start falling from the sky and landing in their backyard:
I was working for Hughes, now Boeing Satellite Systems, for a couple of years doing Y2K remediation. I was writing ground station code that were on VAX/VMS, basically older hardware and software. So that thought did occur to me of satellites falling from the sky. From what I understand, everything worked out ok. There might be only one instance that we know of that was unanticipated when the year roll-over.
I had a teacher in college who'd been part of a project to fix the Y2K bug in the bank he worked at the time. He said to us when talking about project management, that the only project he saw actually finish on time was that one.
The pressure was immense. People don't realise what a huge success story the Y2K situation really was.
I worked at a University that had a 28 story tower. Around 1998 they set the clock in the lift management system to 1 January 2000 and all the lifts lowered themselves to the bottom of the lift wells because they hadn't been serviced for 50 years (or something).
So yes indeed stuff was broken, but it got fixed before the big date.
This sounds apocryphal. If setting the date forward triggered a maintenance warning, it sounds like the date system was working correctly.
More importantly, lift systems are not sensitive to long duration dates. They do not need to be.
This story reflects one of the myths that emerged from the media. Around 1997 they started discovering the topic of embedded systems and that many devices, including lifts, contained "computers." Not understanding the restricted and specialized nature of embedded systems, they then claimed all these systems were vulnerable to Y2K, and that the whole world was about to crash.
Most embedded systems, in lifts and other devices, had been designed in the 1990s. If they used dates, they did not use mistakes from the 1960s.
> Yes, it's true that all software occasionally has bugs. But when all the software fails at the same time, a lot of the backup systems simultaneously fail, and you lose the infrastructure to fix things.
This is especially important if you think about what things were like in the late-90s. As a teenage geek in Northern Ireland I read about the the first tech boom, but locally there was very little exploitation of tech and you were an anomaly if you had dial-up internet at home. There was very limited expertise available to fix your computer systems in the best of time.
Big companies had complex systems, but they could also afford contracts with vendors in the UK mainland, or Eire, to fix systems. It was much more limited for other companies. While people were generally not relying on personal computers too much, small businesses had just reached that tipping point where Y2K would have been painful. As a result, they took action.
I only got to visit the US a few times in the early-nineties and it seemed so futuristic (I got a 286 PC in '93 to use alongside my Amiga). I imagined the Y2K problem as being much more painful in the US, and I wasn't accounting for the distribution of expertise.
One great experience was summer training courses that the government had organised with a IT training company in Belfast. These were free and it was a like a rag tag army of teenagers and older geeks from both sides of what was quite a divided community. The systems we were dealing with were fairly simple. It was mostly patching Windows, Sage Accounting, upgrading productivity apps (there was more than Microsoft Office in use) etc., but the trainer was normally teaching more advanced stuff like networking and motherboard repair so we spent a lot of time on that.
I was a computer operator in the late 1970s and I used to read ComputerWorld which was the biggest enterprise tech "journal" (actually a newspaper) of the day. In the late 1970s they had a number of articles commenting on the need to fix the Y2K problem. So smart people were planning for it well ahead of time. On the other hand memory and storage were so limited then it was probably hard to get permission to allocate two more bytes for a 4-digit year. But people were definitely thinking about it for a long time before the year 2000.
What you're saying here is that there was no crisis.
The fact that people worked on date issues for a decade does not mean the public scare campaign in the late 90s was justified.
You could equally claim airplanes would crash if they weren't refuelled, or people would die if garbage wasn't collected. In those cases, the media is educated enough to know the claims are pointless.
In the case of y2k, they had little understanding of how enterprise IT worked, and lots of businesses who were happy to encourage that ignorance.
Not only was it real, some of the fixes were kludges put in place that would only last for 20 years. But that was fine, because surely we'd have done longer term fixes by then?
Well when most of the bad Y2K code was written they knew it was bad, they just didn't think civilization would last until the year 2000. It was the Reagan years, after all.
I was not working on fixing Y2K issues, but I did notice the impact it had on systems that hadn't been patched. It's the typical IT conundrum, when you do a good job no one notices and you don't get rewarded for doing a good job; the only recognition comes when things fail.
In 1998 we tested the new computers we sold and many failed or gave odd results when the date was changed to 2000. By mid 1999 almost none of the computers had any problems if you advanced the date.
Also one of the major results of the Y2K bug, IT department finally got the budgets to upgrade their hardware. If they had not gotten newer hardware I am sure there would have been more problems.
Finally, in my area the main reason companies failed from IT problems is because of problems with their database, but it turns out their backup are not good or have not been done recently. Many companies tried to be cheap and never updated their backup software, so even if they did backup their data the backup software could really mess things up if it used 2 digit dates to track which files to update.
Things go very bad if you lose Payroll, Accounts Payable or Accounts Receive-able.
What does the science say? I have seen exactly zero studies which claim that the y2k bug would have led to disastrous consequences if action had not been taken.
Compare that to the CFC situation in the 80's. Scientists agree that the mitigating actions we took saved the ozone layer. Or compare it to the current global warming crisis. Scientists tell us that if we do nothing, we will suffer catastrophic climate change.
Media never tells you the truth, but the scientists usually do. So you listen to them.
There were a lot of two-digit dates out there which would have led to a lot of bugs. Companies put a lot of effort into addressing them so the worst you heard about was a 101 year old man getting baby formula in the mail.
The media over-hyped it, though. There was a market for books and guest interviews on TV news, and plenty of people were willing to step up and preach doom & gloom for a couple bucks: planes were going to fall out of the sky, ATMs would stop working, all traffic lights were going to fail, that sort of thing. It's like there was a pressure to ratchet things up a notch every day so you looked like you were more aware of the tragic impact of this bug than everyone else.
That's the part of the crisis that wasn't real, and it never was.
It wasn't a crisis, but it was a real problem that needed to be, and was, fixed in plenty of time. It didn't surprise anyone in the industry as it was well known throughout the 90's it was coming. The biggest problem was identifying what would break and either fix or replace it. Many companies I dealt with at the time humorously did both: they had big remediation projects and as soon as they finished, decided to dump most of the old stuff for shiny new stuff anyway.
Not really. At least not like it was portrayed. The public thought that all computers stored dates like `99` for 1999, so potentially all code that handled dates/times would need to be fixed.
But actually most software uses epoch time or something similar. So the scope of the problem was much smaller than the news implied.
I was involved in several of the efforts at the time including building the communications systems for the "studio NOC" at AT&T in NYC. I started hearing about vulnerable systems about 5 years before 2000 and we were doing serious work on those systems about 2 years before. I predicted (to friends and family who didn't always care to believe me) that it would be a non-event because disruptions would be localized in smaller systems (we were expecting local banks and credit unions). Even I was blown away by how few of those systems had problems. So know when people say Y2K was no big they fail to recognize the work that went into to ensuring it was a non-event.
There's a very current equivalent - if we're good about social distancing, people may talk about COVID-19 the same way.
I had some SGI IRIX machines impacted by the Y2K bug: if you ran an unpatched OS, nobody could login after Jan 1, 2000 0:0:0Z.
One of them, was running calculations 24/7 for a research group at the university and fortunately they were able to stop the jobs in time for an OS upgrade.
383 comments
[ 2.2 ms ] story [ 196 ms ] threadI lived and worked as a software developer through the Y2K "crisis" (although I wasn't working on solving the crisis myself). Everyone was very worried about it. Nothing really went wrong in the end.
Was that because there was no problem? Or because everyone was worried about it and actually solved the problem? I don't think it's easy to tell the difference really.
I think you mean "working ON it". Talking to developers as a broad group from that time wouldn't necessarily produce any useful information.
The person you replied to was himself a developer working in the late 90s. During the late 90s, I talked to a lot of developers, but only a small percentage of them were on Y2K jobs.
So while we were -aware- of the Y2K issue, it didn't impact any of us in a concrete fashion. We would talk about people we knew on Y2K projects, which were mostly mission critical legacy systems.
So it's not inconceivable for devs in the 90s to have only cursory awareness of the -real- issues that the people who worked on Y2K projects were facing and solved.
https://medium.com/@donhopkins/cobol-forever-1a49f7d28a39
For context, most of the devs I worked with at that time finished their CS degrees within the previous 10 years, and were not legacy systems programmers. Doing 2 digit years was inconceivable to them.
That said, if we need to jump start the economy again, maybe we could come up with a fake problem this time. How about we say that 2025 will be a huge problem for many critical IT systems? Or does a timestamp does produce a nice round number in the near future that is a good sell?
edit: There is this: https://en.wikipedia.org/wiki/Year_2038_problem
The OP can only be asking about comparisons to a hypothetical world where nobody dealt with it for critical systems.
However arguably most dates in corporate IT work are involved in some level of forecasting and prediction and future planning.
In reality, starting in 1970 anyone writing an amortization table program for a 30 year mortgage had to work around Y2K. Anyone dealing in any way with the expiration date for a twenty year term life insurance policy had to start caring about Y2K in 1980. Even a mere net-30 business to business payment account either broke or not in nov-1999. Even on Dec 31 1999 it was hilariously charming how people all around the world thought all computers were located in THEIR timezone and thus any real time clock type failures would occur at precisely midnight local time where they live as opposed to where the computer is actually located. Due to the miracle of UTC time anything bad would have happened to our stuff early in the day while I was eating a late dinner, not when the operations center overstaffed during local timezone 3rd shift.
I was working at a telco at the time and we were very worried and overstaffed over Y2K, our stuff was fine, but we were pretty worried about rioters and such if anyone ELSE failed, like maybe the power co. Hilariously the power co people were probably overstaffed over Y2K, despite knowing their stuff was fine, they were likely worried about those telco goofballs failing thus losing SCADA links to their substations, LOL.
In the end it seems pretty much nothing failed anywhere, as I recall. Or the failure rate for that day, was no higher than any other average calendar date.
There was a decent influx of older devs using the media hype as a way to get nice consulting dollars, nothing wrong with that, but in the end the problem and associated fix was not really a major technical hurdle, except for a few cases. It is also important to understand a lot of systems were not in a SQL databases at the time, many were in ISAM, Pic, dBase (ouch), dbm's (essentially NoSql before NoSql hype) or custom db formats (like flat files etc) that required entire databases to be rewritten, or migrated to new solutions.
My 2 cents, it was a real situation that if ignored could have been a major economic crisis, most companies were addressing it in various ways in plenty of time but the media latched on to a set of high profile companies/government systems that were untouched and hyped it. If you knew any Cobol or could work a Vax or IBM mainframe you could bank some decent money. I was mainly doing new dev work but I did get involved in fixing a number of older code bases, mainly on systems in non-popular languages or on different hardware/OS because I have a knack for that and had experience on most server/mainframe architectures you could name at that time.
They did, last week, put in ... haproxy as an SSL terminator in front of the main server, and will test a switch over this week. This was 8 months of foot-dragging for about 3 hours of setup/config, and a couple more hours of testing. When all your clients are hitting a web server, and their browsers will all stop rejecting your certs, things will get ugly fast - as in "your business will effectively stop functioning". It just sounded like "doom and gloom" but... how do you message this effectively? It requires the receiving parties to actually understand the impact of what you're saying, regardless of terms you use.
Advocating for 2 years to migrate off that box.
For executives with limited IT experience I honestly don't know if there is a good solution, other than having them deal with the disaster and having a clear paper trail that points the finger in their direction. They won't make the same mistake twice.
You say "If we don't solve this by X date, we are looking at losing the ability to take in revenue and possible law suits for failure to perform. It will take Y amount of time to accomplish this"
If you don't couch things in terms of time, money, resources, client impact. They will not care. You can say "Hey, it is super bad that we are running Tomcat 7.0.0 there are a lot of security vulnerabilities" and what they will hear is "blah, blah, blah, we can delay this".
Assuming the entire org is using a captive proxy with an installed CA, identify/isolate every upper-level management person's system(s) in some way, and reroute their access to all internal applications through a deliberately horribly misconfigured HTTPS reverse proxy running TLS 1.0. Their modern browser will explode.
"AAAA, sorry, that thing we've been telling you about for months hit us earlier than we thought it would but thankfully it's only hit all the internal stuff"
"Wait so this is what all of our customers will see?"
"Yes, all of them. Nobody will be to reach us, none of our APIs will work, and we will also break our SLAs on every single contract."
---
You probably want to use GPO to disable bypassing the security warning prompt, and/or set up HSTS for your domains beforehand.
We'll be up for a few Y2K bugs at the start of every decade because 'year += year < n ? 2000 : 1900' was such an easy workaround, for n=20,30,40,...
https://www.pymnts.com/news/payment-methods/2020/new-years-b...
Note that the algorithms were known since at least the late 1960s. Storage space even then shouldn't have been enough of a concern. However people still screw it up all the time.
Sadly this stuff still goes on today, just in slightly different ways. Data type and data structure choices are critical, and it isn't as important to me that someone know how to write an algorithm for a RB-tree (or pick your DS), but it is critical an engineer knows which data structure/type to choose for specific use cases. This is one of my biggest complaints with the way a lot of engineering interviews are structured today, they want you to regurgitate known algorithms which is a waste of time and personal memory, find out if the person knows when they'd use a specific data structure and why the rest is easily researched. I mention this because to me the fundamental flaw of the y2k problem was one of data structure and data type choice.
If you want to store only the year, why not store the years since 1900? That way in a single byte you can go all the way up until 2155. If you want to store it bit-aligned rather than byte-aligned why limit yourself to 00-99? 7 bits can store the year from 1900-2027.
If you want to store day + month + year with a two digit year you have 365.25*100=36525~ options. A two byte value can store up until 65535 options, which can hold all days until 2079.
If you are storing your dates as flat strings (DD-MM-YY or so) then you are already wasting such a large amount of bytes that "storage concerns" seem moot. You are already using 8 bytes to store a date that can be stored in 2 bytes, at that point why not use 10 bytes?
String formatting and display seems more obviously affected to me than storage.
Most input systems were punch cards, which were designed to be fixed-length textual input directly from a user at a card punch machine. Teaching users how to map from “years since 1900” to the correct keys on the card punch is a lot more confusing and error prone than having the computer automatically convert from "71" to 1971.
The phone I carry today is hundreds or thousands of times more powerful than the $5,000,000 mainframe I was working around Y2K and the apps on it - most delivered freely - make it far more capable than the raw processing power difference would indicate.
And the code takes space.
You may think the space is trivial, but I remember working for nearly two weeks on a commercial system to save around 15 bytes on a system, and it was worth it.
Then the culture was embedded ... you worked to save bytes, and didn't do things that would cost more bytes. And the systems worked, and didn't need updated, so they persisted.
However some functions from that time (eg ACCEPT, p. 544) do use 2-digit years. I think you have it exactly right, it was about facilitating string formatting/display: ACCEPT took input from the terminal screen. (I think it is, approximately, gets() + strptime(), though I don't see any talk of error handling). Programmers probably just stuffed the result into a record before writing to DASD (disk), optimizing for programmer time and program complexity over storage.
Perhaps 2-digit years for ACCEPT were an optimization for data entry workers?
A modern z/OS COBOL guide [2] has 4-digit years throughout as far as I can see.
Whilst I'm here, I'm also going to say: "VSAM", "PDS", TSO", "LRECL" and "ABEND" (just because I haven't heard those words for a quarter century and now I'm feeling nostalgic).
[1] http://testa.roberta.free.fr/My%20Books/Mainframe/Fujitsu%20...
[2] https://www.ibm.com/support/knowledgecenter/SS6SG3_4.2.0/com...
You're making an assumption in that statement that there's a byte and it's eight bits in length. This was not always true. Punch cards didn't really work that way, and fair number of CPU's didn't either.
Early machines used various decimal systems and 6-bit characters were widespread also. This explains a bit of why the C language and descendants often have native support for Octal literals and Unix permissions are built around three bit groupings.
On a personal note, even in the very early 90's, I also remember using a CDC machine (a descendant of Seymour Cray's 6600) that supported 60-bit words with 6-bit characters. Pressure to move to 64 bit words with 8-bit bytes resulted in dual mode design that could be booted either way.
I looked up my old code. Shave off a bit of the year and:
a full date in 16 bits. (well, in borland C++ 5 at least)and as it was an accounting application, working with the bitfields was so much nicer than ordinal days you would use nowadays.
month and years was all you really cared about anyway when summarising data. in finance all months have 30 days anyway.
https://news.ycombinator.com/item?id=22356373
At the time I was managing a dBase / FoxPro medical software package...we were a small staff who had to come up with Y2K mitigation on our own.
Our problem is we only had source code for "our" part of the chain...other data was being fed into the system from external systems where we had no vendor support.
Thus our only conceivable plan was to do the old:
It worked in 99.9% of the cases which was enough for us to limp thru and just fix the bad cases by hand as they happened. Eventually we migrated off the whole stack over the next few years so stopped being a problem. I'm sure many mitigation strategies did the same....https://en.wikipedia.org/wiki/Year_2000_problem#Leap_years
It taught me a great lesson about results not proving correctness as how you got there could bite you later.
> This method works fine for the year 2000 (because it is a leap year), and will not become a problem until 2100, when older legacy programs will likely have long since been replaced.
The simple reason is that any one company may have been liable for a very small portion, which if it failed, would not have caused much trouble. But that failure combined with many other failures down the entire chain of connected and unconnected software would have added up to something much greater than the sum of parts.
We saw similar stuff happen with Flash, Silverlight (which wasn't as reported a concern since silverlight usage was so low, but I saw it within my company).
The media pressure was a significant reason why every company needed to have a plan to deal with it.
Because they make more money if people are scared and clicking refresh every few minutes.
That said, none of the bugs would have been critical to the operations of the services. Everything was in the billing systems and I think if unfixed it would have been more of a reputation hit than anything.
Also, "begs the question" doesn't mean what you think it means. https://en.wikipedia.org/wiki/Begging_the_question
> In modern vernacular usage, however, begging the question is often used to mean "raising the question" or "suggesting the question".
To you. But to the store that didn't get any money for thousands of sales and potentially went out of business, definitely critical.
Welcome to most enterprise IT shops.
e.g. "what do we pay all these janitors for when this place looks spotless?"
As for your examples, those are all boolean - they either do it successfully or they don't, and if they do, it's what they are paid for, and if they don't, they catch hell. There is nothing intrinsic about IT that differs.
People were talking about Y2K years ahead of time. Lots of changes to code were made. A few little bugs slipped through, but not many, and everyone knew how to fix them. No crisis. Without the many code changes, big problem.
In the end you end up in this perverse situation where you have to wait long enough for the public to understand what's at stake but not long enough that you won't be able to keep things under control. Quite the tightrope trick.
Unsurprisingly, humans are not good at accounting for black swan events, and even less so for averted ones.
Things that are inevitable only when you encompass time spans longer than a human life (it has been approximately one and a half average human lifespans since the previous pandemic) may be predictable at that large aggregate scale, but on useful scales they are not. Or, to put it another way, if you've been shorting the market since 1918 for the next pandemic crash, you went bankrupt a long time ago.
Y2K is only a black swan for those not in the industry, since that one is obviously intrinsically timing based. The UNIX timestamp equivalent is also equally predictable to you and I, but to the rest of the world will seem even more arbitrary if it's still a problem by then. (At least Y2K was visibly obviously special on the normal human calendar.) But I wouldn't claim the term for that; call it a bit of sloppiness in my writing.
Even if every other country has Y2K levels of success containing the Coronavirus, we can still point skeptics at the example of places like Italy to prove it was a real threat.
https://twitter.com/basiccomic/status/1099332074983641094
Just like the crisis we are currently facing in our health systems, it seems unlikely that we would have had enough IT resources to deal with the issues in real-time.
This is one of the cases of a "self-denying-prophecy", much like acid rain. There was an issue, we collectively dealt with it (better yet, we actually anticipated!), and now people are saying that in the end there was no issue.
https://www.bbc.com/future/article/20190823-can-lessons-from...
The Y2K bug did not become a crisis only because people literally spent tens of billions of dollars in effort to fix it. And in the end, everything kept working, so a lot of people thought it wasn't a crisis at all. Complete nonsense.
Yes, it's true that all software occasionally has bugs. But when all the software fails at the same time, a lot of the backup systems simultaneously fail, and you lose the infrastructure to fix things.
Whole businesses would be ground to a halt to stem the mess of bad data and if not prepared they would not have quick fix.
In many cases in the age of apps and web apps we can roll out fixes fairly quickly, but in 2000 that was very much not the case for most situations.
I wonder what the practical mixture of NAND flash vs EEPROMs was. I understand NAND wasn't especially stable back then.
Once upon a time we got somewhere in the neighborhood of 1200 dell workstations and not a single one failed the bios upgrade. 10/10 would do again
The sceptic inside me is curious as to how much they actually accomplished in their effort to fix it. I mean, yes, they spent billions to get millions of lines of code read, but how many fixes have been made and what would be the cost (when compared to those billions of dollars spent) if they weren't fixed at all.
With an auditorium full of others who had barely any programming experience, he was given a crash course in Cobol for a month, and then he had to go through thousands of lines of code of bank software that ran on some mainframe and identify all cases where a date was hard-coded as 2 characters.
Extremely tedious work, but there were tons of fixes to be made.
The problem was real, and the billions that were spent on fixing things were necessary.
EDIT (for clarity): that’s what I would say, and I’m pretty sure many people smarter than I had said it - so I’m just curious what objections made business push forward with the “relatively unskilled” human labour intensive practices instead, because that’s the history we have now.
Also, much of the remediation could not be automated anyway (for example you may not be able to assume the century is '19' for a given date field which can be in the future. You may need to do a comparison to a cutoff date, or you may only have to enlarge the field knowing that the value is supplied from elsewhere which will itself get fixed), so it fell back to people looking at the code, understanding it just enough, making the required changes, and planning/executing the testing.
Remember that big old systems were the low hanging fruit in 1970. The business logic was COBOL and assembler, without things like functions. The system may have hundreds of duplicate routines for things like subtracting dates.
Why are you sceptical about this?
"look at y2k: we spent so much money, and nothing happened!" (so far, so good) "...therefore, if we hadn't, nothing would have happened anyway!!" (uh, based on...?) "...and it was all a massive scam, just like any massive present issue!!!" (ah, there's the underlying reason)
If you asking "would it be better to have done nothing?" then I would have to say No, of course not. What are you thinking?
I worked on fixing Y2K bugs at the time. It was very methodically and planned, and tested on-site at a space agency. If we hadn't fixed those Y2K issues at the time then weather forecasts would have suffered greatly. The cost of that happening? Hard to calculate I think.
Yes, updating JDEdwards AS/400 systems and many a PC was a big project, but i dont recall doing it at that time as being super difficult... frack ist so long ago i cant really recall many of the details other than reporting daily on the number of systems updated.
Also, fuck you google - this was when you were nascent a i converted the entire company to your “minimalist” front page vs yahoo, and a few years later you wasted 3 months of my time interviewing to tell me to expect an offer letter tomorrow and called to then reacind the offer (as i didnt have a degree) and then continuing to contact me for FIVE FUCKING YEARS for the same job that you wouldnt hire me for.
/off-my-chest
None of those people contacting you, or phone screening, were Google employees. They were, or worked for, contractors. It didn't matter to them or to Google whether anybody they contacted was hired.
He also noted that a lot of American firms worked with Indian firms to resolve it. Indian engineers were well suited to the problem, according to him, because they had been exposed to a lot of the legacy systems involved during their studies.
He said an interesting consequence of this was that many American companies concluded they could use lower wage Indian engineers on other software projects so it helped initiate a wave of offshoring in the early 2000s.
I couldn't find the story I heard in the NPR archives, but they do have a number of stories reported right around that time if you want to see how it was being discussed and treated by NPR at the time:
https://www.npr.org/search?query=y2k&page=1
I did find this NPR interview with an Indian official from 2009 that notes Y2K's impact in US-Indian economic relations:
But I think the real turning point in many ways in the U.S. relations was Y2K, the 2000 - year 2000, Indian software engineers and computer engineers suddenly found themselves in the U.S. helping the U.S. in adjusting itself to the Y2K problem. And that opened a new chapter in our relations with a huge increase in services, trade and software and computer industry.
https://www.npr.org/templates/story/story.php?storyId=120738...
This created an exploitable opportunity for the few who knew how to write automated tests.
During the Y2K panic Sun Microsystems (IIRC) announced that they would pay a bounty of ~$1,000 per Y2K bug that anyone found in their software (due to the lack of automated tests, they didn't even know how to find their Y2K bugs)
James Whittaker (a college professor at the time) worked with his students to create a program that would parse Sun's binaries and discover many types of Y2K bugs. They wrote the code, hit run, and waited.
And waited. And waited.
One week later the code printed out it's findings: It found tens of thousands of bugs.
James Whittaker went to Sun Microsystems with his lawyer. They saw the results and then brought in their own lawyers. Eventually there was some settlement.
One of James' students bought a car with his share.
It was money for jam.
this left me at a crossroads. i thought about writing malware that would randomly raise peoples semester grades by 4 points (e.g. C- would became a B+). i thought about mass changing grades. i thought about altering a target group of kids i didn't like. all but the last of the scenarios ended with me getting fingered because I was the smart computer kid. if i didn't touch my grades i would have plausible deniability. i wrote the malware. then i watched office space and decided to think about my actions (and promptly forgot as a horny teenager does). soooo glad i didn't release it because years later i went back into the code and found a leftover debug that would have targeted the only 2 kids in the school who had this letter in their last name.
tl;dr: office space is real
For example, at a former employer of mine, a big justification for getting rid of the IBM-compatible mainframe and a lot of legacy systems which ran on it (various in-house apps written in COBOL and FORTRAN) was Y2K.
In reality, they probably could have updated the mainframe systems to be Y2K-compliant. But, they didn't want to do that. They wanted to dump it all and replace it with an off-the-shelf solution running on Unix and/or Windows. And, for reasons which have absolutely nothing to do with Y2K itself (the expense and limitations of the mainframe platform), it probably was the right call. But, Y2K helped moved it from "wouldn't-it-be-nice-if-we" column into the "must-be-done" column.
Story time: years ago (2010-ish?), we were doing some light web CMS work for a client. Nothing too complex, except we had to regenerate one section off a data feed that we received in a very peculiar custom format.
Great, everything worked, we finished on time and on budget. And then: "oh, and could you also check this other site? We need some minor tweaks on what your predecessors made". That other site was also consuming that data feed, so we took a peek. It ran in PHP3, walled off from everything, processed its own intermediate languageS, and the output was a massive 200+ page PDF (which was then manually shipped to offset print in large quantities). For Reasons, this had to run daily and had to work on first try. There was no documentation, no comments, and no development environment: apparently this was made directly on prod and carefully left untouched.
Needless to say, the code was massively interdependent, so that the minor tweaks were actually somewhat harder. We did manage to set up a development VM though, and document it a bit - but last I checked, the Monstrous Book Of Monsters Generator still seems to be chugging along in its containment.
Sounds like some of the benefits from the Coronavirus crisis, like the increase in working from home.
And don't get me started on Oracle refusing to provide updated y2k compliant backports!
> In reality, they probably could have updated the mainframe systems to be Y2K-compliant.
Legacy systems weren't always mainframe systems and, in any case a “legacy system” is precisely one you are no longer confident you can safely update for changing business requirements.
A change in a pervasive assumption touching many parts of a system (like “all dates are always in the 20th century”) is precisely the kind of thing that is high risk with a legacy systems.
I agree with everything you and the parent said, but there was also hype and hysteria on top of it all - especially among the communities of preppers, back to the landers, and eco-extremist types. I met entire communities of technophobes who declared I was just ignorant or 'a sheep' and refused to believe me when I tried to explain that the industry was working on the problem and had a solid shot at preventing major catastrophe.
Frightening predictions of power plants going offline for days, erratic traffic lights causing mayhem, food and water supplies being interrupted, even reactors melting down were not uncommon.
As always, there is money to be made by exploiting people's fear. This hysteria was considered an irrelevant side show by most educated people, the way I look at the chem trail people or the way I used to view the anti-vaxxers [1], but the unjustified hysteria was real.
[1] Before I learned their numbers were growing and could threaten herd immunity in some areas.
And a few people did take it really seriously and spend the Millennium in some remote place where they would stand a chance of surviving the collapse of Western Civilisation.
https://www.youtube.com/watch?v=x0bBYK-7ZiU
Oh, and Dick Clark melts.
The pressure was immense. People don't realise what a huge success story the Y2K situation really was.
So yes indeed stuff was broken, but it got fixed before the big date.
More importantly, lift systems are not sensitive to long duration dates. They do not need to be.
This story reflects one of the myths that emerged from the media. Around 1997 they started discovering the topic of embedded systems and that many devices, including lifts, contained "computers." Not understanding the restricted and specialized nature of embedded systems, they then claimed all these systems were vulnerable to Y2K, and that the whole world was about to crash.
Most embedded systems, in lifts and other devices, had been designed in the 1990s. If they used dates, they did not use mistakes from the 1960s.
The system was probably designed in the 1970s.
This is especially important if you think about what things were like in the late-90s. As a teenage geek in Northern Ireland I read about the the first tech boom, but locally there was very little exploitation of tech and you were an anomaly if you had dial-up internet at home. There was very limited expertise available to fix your computer systems in the best of time.
Big companies had complex systems, but they could also afford contracts with vendors in the UK mainland, or Eire, to fix systems. It was much more limited for other companies. While people were generally not relying on personal computers too much, small businesses had just reached that tipping point where Y2K would have been painful. As a result, they took action.
I only got to visit the US a few times in the early-nineties and it seemed so futuristic (I got a 286 PC in '93 to use alongside my Amiga). I imagined the Y2K problem as being much more painful in the US, and I wasn't accounting for the distribution of expertise.
One great experience was summer training courses that the government had organised with a IT training company in Belfast. These were free and it was a like a rag tag army of teenagers and older geeks from both sides of what was quite a divided community. The systems we were dealing with were fairly simple. It was mostly patching Windows, Sage Accounting, upgrading productivity apps (there was more than Microsoft Office in use) etc., but the trainer was normally teaching more advanced stuff like networking and motherboard repair so we spent a lot of time on that.
The fact that people worked on date issues for a decade does not mean the public scare campaign in the late 90s was justified.
You could equally claim airplanes would crash if they weren't refuelled, or people would die if garbage wasn't collected. In those cases, the media is educated enough to know the claims are pointless.
In the case of y2k, they had little understanding of how enterprise IT worked, and lots of businesses who were happy to encourage that ignorance.
Except we didn't:
https://en.wikipedia.org/wiki/Year_2000_problem#On_1_January...
I think it's a great lesson in the failings of the public imagination and should serve as a warning to not give into moral panics.
Some historians seem to think that it was a real crisis in which the US pioneered solutions that were used across the world: https://www.washingtonpost.com/outlook/2019/12/30/lessons-yk...
Also one of the major results of the Y2K bug, IT department finally got the budgets to upgrade their hardware. If they had not gotten newer hardware I am sure there would have been more problems.
Finally, in my area the main reason companies failed from IT problems is because of problems with their database, but it turns out their backup are not good or have not been done recently. Many companies tried to be cheap and never updated their backup software, so even if they did backup their data the backup software could really mess things up if it used 2 digit dates to track which files to update.
Things go very bad if you lose Payroll, Accounts Payable or Accounts Receive-able.
Compare that to the CFC situation in the 80's. Scientists agree that the mitigating actions we took saved the ozone layer. Or compare it to the current global warming crisis. Scientists tell us that if we do nothing, we will suffer catastrophic climate change.
Media never tells you the truth, but the scientists usually do. So you listen to them.
There were a lot of two-digit dates out there which would have led to a lot of bugs. Companies put a lot of effort into addressing them so the worst you heard about was a 101 year old man getting baby formula in the mail.
The media over-hyped it, though. There was a market for books and guest interviews on TV news, and plenty of people were willing to step up and preach doom & gloom for a couple bucks: planes were going to fall out of the sky, ATMs would stop working, all traffic lights were going to fail, that sort of thing. It's like there was a pressure to ratchet things up a notch every day so you looked like you were more aware of the tragic impact of this bug than everyone else.
That's the part of the crisis that wasn't real, and it never was.
But actually most software uses epoch time or something similar. So the scope of the problem was much smaller than the news implied.
There's a very current equivalent - if we're good about social distancing, people may talk about COVID-19 the same way.
One of them, was running calculations 24/7 for a research group at the university and fortunately they were able to stop the jobs in time for an OS upgrade.