Ask HN: Ever worked with a service that can never be restarted?
I'm currently working with a legacy system. One element of it has a loaded config in memory, but the physical config was accidentally overwritten and there are no backups. In addition the source code for this compiled binary has also been lost in the mists of time.
The service has current up time of 55 Months. The general consensus therefore is that as long as it is never restarted it will continue to perform its function until a replacement can be put in place. Which seems a little fatalistic to me...
Has anyone experience of doing something sensible in a similar situation?
200 comments
[ 354 ms ] story [ 4755 ms ] threadAs is, this is all relying on the HW not giving up and the UPS working in a power failure!
the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...
For servers I expect that they would have redundant inputs, that you can easily hot swap it.
The second link had a video on how to do that. With multiple ways to achieve a power re-route, including a tool to easily splice into the wire. The easiest technique was to just use a sheath to touch the metal prongs on the cable.
But the surprise for me was the power strip technique, where he plugged in his battery source into one of the 6 outlets, thereby simulating a power input. I didn’t even know that was possible.
This is true, but there is still a real risk that the dual power input on the server fails when you actually do the swap.
For example, the second PSU may not work, or it may have failed in the past and then be forgotten about, or connector or cable internally may be flaky, or the component that combines power from both PSUs may fail (although I think that's quite rare).
If it's a really critical service, those risks may be unacceptable.
It's somewhat interesting how this company can sell a product which obviously violates pretty much any electrical safety regulations (a plug with exposed prongs that can become hot). I'm guessing this is sold as a kit, not as a complete product.
In these cases it's probably worth doing it though.
https://media.screwfix.com/is/image//ae235?src=ae235/38040_P...
FWIW I do not endorse doing this, just stating that its possible and would work, it's not electrically safe and is dangerous for not only the hardware but for you too.
https://youtu.be/-G8sEYCOv-o
Pretty ingenious, tbh. Though I'm a bit concerned about how much you end up handling energized plugs.
It is then be a task to reverse engineer the coding file from the config data structures.
The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely.
First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM because there might be vMotion or some kind of live-migration in place meaning hardware failure might not actually bring the service down.
Next thing is you absolutely have to plan for failure because the one thing that I learn as I learn more about computers is that they're basically working "by accident", so much error correction goes into systems that it absolutely boggles my mind and failures still get through. So, it's certainly not a question of "if" but "when" and plan for it being soon.
Now, the obvious technical things are:
* Dump the memory * Grab any files it has open from its file-descriptors (/proc/<pid>/fd/) Your config file might be there... but somehow I doubt it. Attach a debugger and dump as much state as possible
Be sure to cleanly detatch : https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_22.htm...
Don't use breakpoints! They will obviously halt execution.
If it was my service I would also capture packet's and see what kind of traffic it receives most commonly; and I would make some kind of stub component to replace it before working on the machine. Just in case I break it and it causes everything to go down that depends on it.
But, this is a horrible situation, reverse engineering this thing is going to be a pain. Good Luck.
So true. Thanks this is good advice. As it happens it is a physical server.
I'm not actually mandated to do anything except replace it with a comparable service but this makes me feel like I'm racing against time!
Mind you if things are that bad then maybe I'd follow my own advice (in another comment) and stay well away from the currently running thing.
Using this Shadow Ghost Ship you can build a copy of tcpdump that you can just scp over and run in place. Don't touch the package manager, /etc, /var, anything!
So whatever is done cannot risk the running process. So personally I'd be loathed to try anything that touches the running process - even if they seem low risk.
Edit: One thing I would do is make it clear that building a replacement and keeping this thing running are different responsibilities and if you are doing the former then you really shouldn't be held responsible for the latter (politics I know).
Attaching a debugger can cause multi-threaded processes to behave differently and/or deadlock if you are unlucky.
Also, some closed-source software components deliberately crash if they detect a debugger and since you don't have source code for the binary, there is a risk that one such component was used during development.
The company may have failed but when the proverbial truckload of shit hits the proverbial supersonic fan there will be tremendous pressure to find a culprit. A small peon might find himself/herself outnumbered and outgunned.
This should reduce risk a bit because you are not experimenting and creating your procedure on the fly when you do the actual thing. Instead, you are applying a procedure that you have rehearsed, and you have gained partial confidence that it won't disturb a similar running binary.
The mock doesn't have to be a perfect reproduction of the real binary, but it should do similar things where relevant. For example, if the service listens on a TCP port, so should the mock. You might not have to code the mock from scratch; maybe some existing software is similar enough. Or maybe you can even run a non-production copy of your real binary, if you are really confident you can configure it to not interact with anything it shouldn't.
Surely there are options, have you tried de-compiling the source from the binary?
>but the physical config was accidentally overwritten and there are no backups
Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old legacy systems it's fairly likely someone will have a copy somewhere.
I think you mean "older devs that are still with the company" but I was once contacted by a former employer for exactly this: they had lost source code for the billing system. I didn't have the source code so that was easy to answer. Developers that have stolen intellectual property from former employers may want to consider carefully how they answer such questions...
Also, I was never supposed to have had access to this code whilst I was employed so I did wonder if there was more to it than just "oops we lost it can you help?". Perhaps they were trying to chase down a leak or something? I never heard back though (they survived).
IANAL, but a hold harmless agreement would be a simple CYA that I'd do if I was personally in this situation.
If 'going down' is an event governed by chance (e.g. power failure), then it does not matter if it has been up for 55 months or 55 minutes.
https://en.wikipedia.org/wiki/Gambler%27s_fallacy
Think the probability of the machine shutting down in the next minute is independent of age, but it shutting off in the next 100 years is certain. There’s a line or curve that should indicate probability of shutting down before OP writes the replacement.
See eg https://en.wikipedia.org/wiki/Bathtub_curve for a common way to fail. Or https://en.wikipedia.org/wiki/Reliability_engineering in general.
You are right about some specific kinds of spontaneous failures. (Though not sure if external power failures are distributed that way.)
The fact that the server has been running uninterrupted for 55 months is a strong clue that the power to this system is robust, not subject to "random" power loss scenarios like one would expect at a typical residence.
I don't remember random power losses in typical first world residences after perhaps the 1990s?
Less likely
https://en.wikipedia.org/wiki/Lindy_Effect
> The Lindy effect is a theory that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining life expectancy. Where the Lindy effect applies, mortality rate decreases with time. In contrast, living creatures and mechanical things follow a bathtub curve where, after "childhood", the mortality rate increases with time. Because life expectancy is probabilistically derived, a thing may become extinct before its "expected" survival. In other words, one needs to gauge both the age and "health" of the thing to determine continued survival.
Also, unless the server is in a data centre with some serious redundancies, (and even then...) the power is guaranteed to go down one day.
A computer program that is freewheeling along, by contrast, is not sustained by but dependent on external factors (software, hardware, electricity), all of which are prone to fail and some of which are more likely to fail with age.
The question is which effect dominates at a particular point in time.
But I guess the bigger problem here is the missing config - as long as you still have the binary of the application, you should be able to restart it.
We started working on solution to run parallel to the existing one, which would receive shadow traffic so we could observe behaviour, find all edge cases and put them in a testsuite for the new solution. After we where confident that our testsuite contained most of the important behaviour we switched traffic to our new solution, keeping the old online just in case we needed to switch back.
Key is monitoring and learning the expected behaviour of the connected systems, to sense if that behaviour is not as expected and being able to act upon that as soon as possible.
Just another perspective on that server message. This server might be actually running a real-time legacy interface to one of your biggest customers who pay a 6 figure premium to keep this old thing running, just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there.
I'm not saying this happened here, but it happens a lot more than any of us want. Just keep in mind there might actually be a good reason for the warning, although I would object to the wording. First head the warning. Then figure out why it is there.
The questions I would ask are: what is the insurance if the system fails due to natural causes (hardware failure, power outage, you name it)? And what is the cost/benefit balance of having this unpatched security hole connected to the internet (which was the reason we discovered this).
If a system is so important that such a warning is deemed appropriate, imho its just laziness and carelessness from the warning's writer and they are shifting their problem onto the next person that stumbles upon the system.
At least give some information on why it is important and at what time or under which conditions the warning can be considered expired. Just like commenting your code is important because you never know who is having to make sense of it in a few months time (especially true if that person turn out to be you :)).
In answer to your first question. There might not be an insurance against the system failing. The honest answer to what to do then, because someone always needs to ask is: Panic! It might not even be unrecoverable, just really expensive and time-consuming to do so.
The follow-up question would be: Yes it is a risk, yes it is a possible disaster. What do you want to do about it? If your answer is anything more than: We should not have taken on that risk in the first place or lets bet the companies future on the fact that we can fix this. Then it might actually be interesting to listen to. It seems from your comment you resolved it.
As for messaging. I do not agree you should explain the situation on the server. Just let people know they should not touch this ever. As soon as you explain why people will assume their reason to do so will trump whatever reason you gave. If they need to think of all possible disasters that might happen it has more impact than the one you can describe.
Messages like "Before doing anything on this server contact Bob" will eventually lead to Bob receiving a message: "We have done this or that. Just letting you know, but it was after office hours" which he will probably see rushing into the office in the middle of the night because the server is not working anymore. The other type of message: "Don't reboot this server ever! For more information contact Bob" Bob changed because he spend a considerable time saying no and explaining the situation to the sysops team, their manager, their manager's manager, etc. who all thought their priority must trump Bob's. Bob might still be working for the company. He might have tried the better part of his career to get this stain resolved. Nothing bad about Bob.
My experience with migration projects is that they can drag on in time, and all the while this system is just itching to go down due to a power failure or some other issue.
If it’s vm take a snapshot periodically, I suspect it’s not.
Try a p2v which converts it to a vm on the fly, and leave the vm off and periodically rerun the p2v
Don't touch the server, if anything happens, you'll be held responsible.
The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets.
It’s still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.
In this case the input space is small and you want a quite dense mapping from input to output. I do not think ML applies.
The problem is that the solution space grows super exponentially, and if you need to find an exact one, then the number of samples too.
No. Not even close for any general purpose I/O.
This reminds me of Team America [1] ... or maybe idiocracy [2] :-p
[1] https://www.youtube.com/watch?v=DIlG9aSMCpg
[2] https://www.youtube.com/watch?v=kIZ9YuPm_Ls
This paper [0] is a pretty decent survey on the field and [1] is an excellent article on the topic at a high level, written by a team currently working in the space (synthesising smart contracts).
[0] https://arxiv.org/pdf/1802.02353.pdf
[1] https://medium.com/@vidiborskiy/software-writes-software-pro...
Do you think we can put some crypto in there, too?
Too subtle for HN but I don’t mind.
No, plenty of us got it. It's just not funny right now.
A person is trying to find a real solution to a real problem they have. Every comment in this thread represents a potential hope that someone knows how to fix it. When someone comes in to a serious thread that is spitballing ideas and tries to make it about them by cracking jokes and being fun and clever, it's annoying. I know this because for about 8 years I was the guy trying to crack jokes in these threads.
This is not the time to make some kind of point about the state of modern software engineering - do that in a thread where people are bullshitting, not trying to work a problem. It's not ever really funny. It's noise. It doesn't help anything. It's not too subtle, it's just obnoxious.
Watch as I come up with a random reason you're wrong.
The OP could see it and laugh and that could possibly lighten their day and help them relax.
Wait, no. That's positive, online we're only supposed to bash men, hitler, and take everything in the most negative light possible. I should have instead judged you as a hateful person who has a horrible life and no friends.
https://news.ycombinator.com/newsguidelines.html
Just like your reply.
Besides that: OP is fucked and left with a huge pile of technical debt. While his intention to help and find a solution is noble, one should face the truth: One way or another this system will go down and nobody will be able to restart it. Just leave it where it is and start building a replacement that is well documented and has backups. Any time spent on saving this system is thrown away. So: This is totally the time to make fun. Call it gallows humor or whatever but stop whining.
I've not looked for a long time but you could potentially snapshot the running process and restore it. Obviously do your reading extensively first though.
(Update re the downvote: You can't tell me that uptime isn't a little bit funny. I almost did a spit take at this. As someone who sees and thinks an uptime of 150 days is way too much this number just shocked me, that's all)
Find an expert who has masses of experience who can consult on it.
This isn't a good time to be learning and testing those lessons.
Doing even minor things that could distrupt the execution for any time (attaching debugger, installing packet sniffer that blocks network traffic for a sec or cause a packet lost, etc) could deadlock the application.
Wouldn’t risk it.
You have exponentially more to lose than to gain by experimenting with this at all. In this situation you should convey the risks, let the stakeholders decide if/who is going to address this, as it's their risk to take.
1. That gives you a pretty good chance of attaching a debugger (to the offline memory file) and extracting the config from memory without touching the running system.
2. You are safe in case things turn awful, which seems likely.
As for the source code, if it is an interpreted language like Java or Python or C#, your chances of recovering a fully-working source code tree are pretty good. At the very least, it'll help with understanding what the system does.
For C / C++, buy Hex-Rays IDA and Decompiler. That's what the pros use for reverse engineering video game protections, breaking DRM, etc. So that tool set is excellent for getting a overview of what a binary does with the ability to go into all the nitty-gritty details if you need help re-implementing one detail. Plus, Hex-Rays can actually propagate types and variable names through your disassembly and then export compile-able C source code for functions from your binary.
My colleagues told me that it was still there 2 years ago.
First of all you are there to build a replacement service. As long as you do that in the allotted time what happens to the old should not bother you. Focus on your part of the solution.
The fact that you do seem to worry suggests you need something from the old service. Do you have a good design for the new service? Are there details of the old service that you do not know about? It won't be the first time a company has an application that runs complex calculations, but nobody knows how it is done. We actually built black-box applications in the past to replace calculation modules on obsolete hardware.
If you need information from the old system you may have to resort to cloning the input and output preferably on the network level so you don't have to mess with the existing service.
If the replacement service is years out you might suggest to the company to put in a red team. Another person or persons who work on finding out how the service works by running a duplicate on another system, rent it if necessary, and poking at it seeing if they can get comparable results quicker than the replacement service. But that is not you.
There are a couple of things you need to have in place before going live. Make sure you can clone input and output so you can run the old service next to the new one for a while. You don't want to stop the old service the moment you go live, because if anything goes wrong you need to be able to switch back. Even than starving the old service of input might have adverse effects, so cloning is better. If the old service uses external resources, like a database, files, etc. make sure you do not interfere with file locks, sequence numbers,etc. when running in parallel.
1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now
2. Remind them that it will go down at the worst possible time.
3. Ensure your name is attached to these two warnings.
4. Promise yourself you wouldn't run your business this way.
5. Get on with your life.
- Make a plan of exploratory steps that could trigger a failure, in order of increasing risks.
- Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step.
Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments):
- Logging in to the server at all.
- Splicing something into the network switch.
- Cloning filesystems or disk images.
- Cloning the process memory or system memory.
- Cloning the image or other things if it's in a VM.
- Running strace gdb, or packet tracing.
- Swapping the power source live.
These days, it is often possible to transfer a full working system into a VM on a more modern, powerful machine, and after doing that, it's a great relief to everyone because it's no longer hardware dependent.
I've done that with some legacy systems that were originally on bare metal, and are now still running nearly 20 years later in reliable VMs, with no change to the running software. Usually running faster and with more memory doesn't break a working system.
But doing it on a service that can't be taken down even for a moment is quite an interesting adventure! :-) It is possible, but technically challenging, as long as you can obtain disk and memory images from the live system and then capture changes fast enough to perform a hot transfer.
I’ve been in similar situations, but never without an immediately obvious solution.
That system WILL FAIL. Even as we speak the time-to-failure is shrinking.
Even if one of the solutions described below actually works, you won’t get 100% recovery.
I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When they tried, it didn’t work. Nobody had actually tested recovering from a backup.
Grit your teeth; cover your ass; and get on with it. The clock IS ticking.
I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good.
Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe, it will crash.
After sufficient CYA steps are taken, don’t offer any help on this problem unless explicitly asked to do so. And even then wait until there’s actually desperation. Feel free to think about it so you have some ideas when the time comes, but if this service isn’t something you are responsible for then stay away.
Or else it will be a “you touched it last you broke it” regardless of what actually happened. And if you make too much ruckus about it, they’ll say “why didn’t you do anything to fix it??”
When shit hits the fan if the company wanted to throw you under the bus they could delete them remotely from your machine or work mailbox and then pretend the warnings never existed.