Well, every six months or so I review my various VPSes and I usually find at least one I'd forgotten about, so unless I'm unusually forgetful I'd say "yes" :)
Seen it a few times. Company acquires another company or an employees leaves. And then you find out years later during an office refit that the PC under their desk was running a critical business process.
Once I was an intern at an air force base. In my cubicle, stuffed into a corner, was an old SGI Octane workstation. It was absolutely covered in dust and looked neglected. Its fan was very annoying to me. One day, I unplugged the machine because I figured that it was completely purposeless. Less than 2 minutes later, someone I had never seen was standing at my desk, berating me for turning off a server that ran a set of critical data reduction processes for the nearby wind tunnel.
After that I added a sign to the machine. "DO NOT TURN OFF UNDER ANY CIRCUMSTANCES." And made sure I wore headphones to block out the sound of the fan.
We started a corporate project in BigCorp. You know: Once a corporate software is installed, you claim you're in charge of it, you can hope to get the budget for it, and you're competing with the other European teams of Big Corp. So you install as much as you can and pretend to have been delegated by BigCorp's headquarters. So we installed Artemis in a VMWare on a desktop and got it up running in the first 16 hours of the team ;)
A month later, 4 internal customers on board and 5 other apps installed, we get a new technical expert. The machine was there. He checked. No Weblogic server running. No tomcat. No database software. Not even Microsoft Word. Clean machine. Format, reinstall...
The day after, Corporate gave us the budget for a server room. \o/
Intriguing question; I would think that a certain percentage of servers might have survived their owners post-mortem. Of course there could be many additional reasons for abandoned servers.
Not really "doing anything", but related: I had to set up a voip server for a test project in 2003, which was supposed to be terminated half a year later. Being a test project, I was advised not to document anything formally, just to place it in the rack and ignore everything else. I left this job soon after.
12 years later, the phone number still is working, even though the companies phone system has been relocated to another room and was replaced by another vendor. No idea how this happened.
It could very well be that somebody migrated all the relvant phone numbers off the test system, and chose to preserve those numbers where they didn't know what they were needed for. Might be cheaper than risking losing functionality that somebody depends on. (Especially if the one doing the migration didn't know it was an abandoned test system).
I work at an ISP / housing / colocation company, and occasionally hardware goes missing (nobody knows where it is anymore, it's not where it's supposed to be). Maybe some of them are broken, some might be stolen, but I'm fairly sure others are still running, not serving any purpose.
And from time to time we stumble over some virtual machine (or even phyiscal server) where nobody knows anymore what it's supposed to do; the standard procedure seems to be firewall it off, wait for a few weeks or months to see if anybody complains, and if not, shut it down, maybe archive it.
Same experience here.
Servers get ordered, racked, provisioned and left powered on waiting for a sysadmin to make use of them... which quite often doesn't ever happen, because team priorities change over time, people move on/get fired, entities go through reorgs, etc.
They sit there idling and unattended, burning power and disks, until some script kiddie finds whatever default root password was used or how to exploit some random apache/ssh flaw.
At that point the possibilities are endless: bitcoin miners are quite unnoticeable in most environments, but DDOS/spam zombies, proxies, bittorrent seedboxes, botnet C&C, "warez" and http servers serving drive by exploits are fairly common.
Protip: ask your datacenter provider to power your servers down (be it VMs or dedicated gear) after racking them up. Powering them back up when you really need them will only take you a minute and you'll save big on power, bandwith, security and peace of mind.
There are many (possibly apocryphal) stories about servers lost in ceilings, locked closets, etc that just kept on serving, sometimes critical services.
I think we as an industry have gotten much better about this. In the old days, as small minicomputers and micros expanded into less and less technical businesses, wiring standards and server room design guides were not well-known or followed.
Often, people just sort of winged it. Some employees are more naturally methodical, have better memories, and are longer-tenured than others. Also, many of the stories are from universities, where long-term thinking isn't guaranteed (but embarrassing stories have always been popular to share!).
A quick DDG finds this story from the University of North Carolina:
I know of one interesting story here. Early on in (what would eventually become a very successful) tech company's life, they acquired some collocated space for free thanks to a friend (call him Brad) of an early employee.
Eventually Brad left that particular DC operator, leaving the startup with no inside contact - but the servers stayed up for years to come (and the company was never charged for the resources).
These machines were eventually "replaced" with newer hardware in a nearby facility, but the DC operator has no idea where these old machines are physically located within the facility (thus cannot remove them), so they remain active to this very day, sitting idly by...
MOTES have an average lifetime of 5 years. Unless they are linked to a power grid and stable internet. Most of these are either broken/frozen/unreachable/out of power.
We actually study this special case in Embedded systems engineering.
He could, except this quote predates Bitcoin by at least five years. Oldest reference for this page in the wayback machine is in 2003. The quote may be even older than that.
And I think the downvotes are uncalled for, the general idea is excellent, even if Bitcoin mining is iffy in a corporate setting (very bad if you keep them, and the finance people won't know what to do with them, plus their tax treatment is not trivial).
Been there. Except in my case it was two 8 story office buildings.
Stuff was moved in hastily from an acquired company when there was no room in the server room, hooked up because important (mail server), and subsequently forgotten while departments where shuffled around the building and employees left over a period of many months.
A week after I started the server started having issues. And nobody knew where it was... Finally found it in some storage room.
That still won't help you find where the machine is located though. With cables you can presumably just follow the cable when you realize disconnecting one stops the pings
Wardriving software like kismet will show your the RSSI of the AP. Is it possible with stock hardware to determine the RSSI of connected devices? Then it is just the hot/cold game. I'm sure actual radio triangulation/direction finding would be possible as well but I'm not sure how difficult that would be.
You could have an electrician identify circuits and start throwing circuit breaker switches to achieve the same result? (obviously might be a bit chaotic, but it'd be a good time to test your UPS infrastructure at the same time, right?)
This week I got a support call from one customer: "hey, we have a maintenance scheduled for our Dell servers, is it ok if we shut down the two pre-prod servers?" "what pre-prod servers?" [all the non-production servers were virtualized about 4 years ago] "well, there are two servers here, with pre-prod labels and running Linux; no applications are running on them"
So I guess 4 years ago somebody migrated the servers to a virtual environment and then just forgot about them. The IPs were migrated to the VMs and these servers were left without an IP address on the network connections, so apart from going to the data room and checking every machine (what they did during the maintenance) there was no way to find them. They were still up and running after 4 years.
Would an arp-scan work or checking the arp cache of switches for MAC addresses that weren't tied to an IP address? Presumably the servers attempt[ed] to get an IP address at some point (like broadcasting DHCPdiscover or some such) - wouldn't it keep operating if an IP weren't acquired?
I am sure there are tens of thousands if not more. Server admins are very risk averse and don't tend to turn off machines just to check if someone's using it.
And in large organizations, it's hard to keep track of who's using what and if they still need it.
Imagine working at a hospital or a bank, emailing people about a server, and everyone says it's not being used. Then you turn it off and something critical gets broken at 3am a week later resulting in an emergency. Who gets blamed, you or the people who forgot the server was being used? The people who set it up may not even be working there anymore.
Yes - in my network admin days back in 2007, we were decommissioning our datacenter as we'd virtualized everything and as we were gutting the room we found our PABX/Voicemail server under the tiles, happily running away with OS/2. Nobody ever knew where it was and it had been installed 13 years before. The floor was on a UPS and generator so the power had never been disconnected.
It was happily chugging away, running our shitty phone system, and hadn't been restarted in 10+ years.
Edit: This also happened to me at a utility company I worked at. We had a server that ran some critical calculations for us but nobody could recall where it was physically located. It's way worse now that everything's virtualized - the cruft just sits there for years until you suddenly run out of resources and start looking closely.
The usual method is to get the device's MAC address, use that to track down/identify the switch port it's connected to, then physically trace the network cable that's plugged into that switch port.
Many/most servers (or the ones I've dealt with the most, anyways) have an LED that can be activated remotely -- i.e. through IPMI, ILO, etc. -- but I can't remember what's it called at the moment. It can be very handy when you're remote and need to be able to direct someone (physically present) to a specific server.
You have attempted to access a blocked website. Access to this website has been blocked for operational reasons by the DOD Enterprise-Level Protection System.
There isn't a separate ping protocol. Echo packets are part of the Internet Control Message Protocol (ICMP). ICMP echo packets are used by both ping and traceroute.
In some sense, traceroute does often give you some location information.
However, if you're talking about GPS or other geolocation information, even now if a machine supports IPv4 and/or IPv6 and isn't a phone, it likely doesn't know any geolocation information about itself. No provision for geolocation information was provided in the ICMP echo or echo response packets.
Prior to opening the comments I thought no way somebody recently found OS/2. Not to one up you or anything, but we just found over a dozen (~15 I think) OS/2 machines in our environment two weeks ago..
Many years ago I did a job for a big corporation with a big IT department. Servers were in the data center of course and to get one you had to fill out a lot of different forms.
When the project was on fire we were allowed to bring in our own machine and use it as a server.
Later, when the project ended no one bothered to go through the whole formal process to get the machine out of the premises again. The machine was just left there connected to the network.
In the months after the project ended different departments moved into the building. Different people with different tasks. Yet the machine stayed connected. I guess no one had the courage to shut down a machine they didn't know about.
For a year or so I still used to check occasionally if I still could log into the machine.
It disappeared eventually but unfortunately I don't remember how long exactly it lasted.
We came across one at work (NASA) a few years back, a server running in a far off room that no one remembered anything about. It had probably last been used by a grad student at least a decade ago, but was still up and running and doing, well, something. We eventually had it removed and turned the room into a collaboration space.
Yes, it happens. From the small company to a corportation, those can be: vms, old servers, racks of servers...
Having a perfect CMDB for hardware and following all procedures step by step could identify / prevent cases like that. But in reality, in normal environment it's going to happen.
My belief is that the only way to make the perfect CMDB software is for your company to write it itself. Every CMDB system winds up being obliterated into twisted machinations of the original vendor's object model to the point where the vendor can't even help their customer anymore. I've witnessed CMDB implementation projects at least a dozen times and have seen 0 actually succeed at what the customer really wanted to get out of it (because people are sold unicorns and puppies in the ITIL model that no vendor can actually deliver because it's theory ware). Almost all CMDBs or equivalent system of records database has been RDBMS based in my experience on the enterprise IT side. One exception is that Facebook has their own that's not from some random acronym vendor used with clients being pluggable such as from Chef in place of the provided ohai.
My first proper software development job was to consolidate the 7 different CMDBs that the company had - they were an ISP/hosting provider who had done a rapid succession of acquisitions, all of which came with their own database of hardware.
Some of the data was so bad that we ended up having one guy who's job was to physically audit what hardware was really in the data centres, and then attempt to reconcile that with the CMDB data.
I've always wanted to intentionally setup servers in hidden places just to see how long they stay online. For example, putting a Raspberry Pi in the back of a rack in a server closet. Maybe run a Tor relay on it or something like that.
At a place I worked we had a service that relied on a cron job running a script every hour. One day someone wanted to know where the server was so he could connect an external hard drive to it and copy some files in. Not a single person knew where the server was or even what it's IP address was. Since it had been setup years ago, everyone who had worked on it had since left the company and no one ever documented it. So somewhere either in our office or in one of three public clouds we used was a server happily running a script every hour that no one could locate or stop. We eventually found it when we moved offices and a super old Dell box in a tiny dusty closet was unplugged and the script stopped running. Now I always document the physical locations of services too.
Years ago I shut down a start-up, and as part of some business partnership, we had a physical server hosted in some big company's datacenter. They forgot about it, and I didn't have time or incentive to drive over there to fetch one old machine. So it just kept running there for a couple years, at first just continuing to serve my failed company's website, and then doing nothing, other than serving as a personal download proxy for me. That lasted for about 2-3 years until they finally found it and shut it off, so I took it home.
My 1996 homepage is still online under its original URL. Which likely means the system hosting it is still online. Except the original ISP no longer exists, nor does the ISP that bought the original ISP.
Next year I'm going to find that system on the 20th anniversary of that page.
112 comments
[ 3.2 ms ] story [ 201 ms ] threadAfter that I added a sign to the machine. "DO NOT TURN OFF UNDER ANY CIRCUMSTANCES." And made sure I wore headphones to block out the sound of the fan.
A month later, 4 internal customers on board and 5 other apps installed, we get a new technical expert. The machine was there. He checked. No Weblogic server running. No tomcat. No database software. Not even Microsoft Word. Clean machine. Format, reinstall...
The day after, Corporate gave us the budget for a server room. \o/
12 years later, the phone number still is working, even though the companies phone system has been relocated to another room and was replaced by another vendor. No idea how this happened.
It could very well be that somebody migrated all the relvant phone numbers off the test system, and chose to preserve those numbers where they didn't know what they were needed for. Might be cheaper than risking losing functionality that somebody depends on. (Especially if the one doing the migration didn't know it was an abandoned test system).
Edit: Now that I think of it, I 've got 3 for 2 different clients.
I work at an ISP / housing / colocation company, and occasionally hardware goes missing (nobody knows where it is anymore, it's not where it's supposed to be). Maybe some of them are broken, some might be stolen, but I'm fairly sure others are still running, not serving any purpose.
And from time to time we stumble over some virtual machine (or even phyiscal server) where nobody knows anymore what it's supposed to do; the standard procedure seems to be firewall it off, wait for a few weeks or months to see if anybody complains, and if not, shut it down, maybe archive it.
They sit there idling and unattended, burning power and disks, until some script kiddie finds whatever default root password was used or how to exploit some random apache/ssh flaw.
At that point the possibilities are endless: bitcoin miners are quite unnoticeable in most environments, but DDOS/spam zombies, proxies, bittorrent seedboxes, botnet C&C, "warez" and http servers serving drive by exploits are fairly common.
Protip: ask your datacenter provider to power your servers down (be it VMs or dedicated gear) after racking them up. Powering them back up when you really need them will only take you a minute and you'll save big on power, bandwith, security and peace of mind.
I think we as an industry have gotten much better about this. In the old days, as small minicomputers and micros expanded into less and less technical businesses, wiring standards and server room design guides were not well-known or followed.
Often, people just sort of winged it. Some employees are more naturally methodical, have better memories, and are longer-tenured than others. Also, many of the stories are from universities, where long-term thinking isn't guaranteed (but embarrassing stories have always been popular to share!).
A quick DDG finds this story from the University of North Carolina:
http://www.bradleymonk.com/wp/weve-lost-server-54/
USENET archives would be a great source for more of these.
Eventually Brad left that particular DC operator, leaving the startup with no inside contact - but the servers stayed up for years to come (and the company was never charged for the resources).
These machines were eventually "replaced" with newer hardware in a nearby facility, but the DC operator has no idea where these old machines are physically located within the facility (thus cannot remove them), so they remain active to this very day, sitting idly by...
We actually study this special case in Embedded systems engineering.
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
And I think the downvotes are uncalled for, the general idea is excellent, even if Bitcoin mining is iffy in a corporate setting (very bad if you keep them, and the finance people won't know what to do with them, plus their tax treatment is not trivial).
Stuff was moved in hastily from an acquired company when there was no room in the server room, hooked up because important (mail server), and subsequently forgotten while departments where shuffled around the building and employees left over a period of many months.
A week after I started the server started having issues. And nobody knew where it was... Finally found it in some storage room.
So I guess 4 years ago somebody migrated the servers to a virtual environment and then just forgot about them. The IPs were migrated to the VMs and these servers were left without an IP address on the network connections, so apart from going to the data room and checking every machine (what they did during the maintenance) there was no way to find them. They were still up and running after 4 years.
And in large organizations, it's hard to keep track of who's using what and if they still need it.
Imagine working at a hospital or a bank, emailing people about a server, and everyone says it's not being used. Then you turn it off and something critical gets broken at 3am a week later resulting in an emergency. Who gets blamed, you or the people who forgot the server was being used? The people who set it up may not even be working there anymore.
It was happily chugging away, running our shitty phone system, and hadn't been restarted in 10+ years.
Edit: This also happened to me at a utility company I worked at. We had a server that ran some critical calculations for us but nobody could recall where it was physically located. It's way worse now that everything's virtualized - the cruft just sits there for years until you suddenly run out of resources and start looking closely.
ah yes, the good old "responds to pings but where is it?"
It was a silly hack that turned out to be clever and incredibly useful.
"[USSC logo] APPLICATION BLOCKED
You have attempted to access a blocked website. Access to this website has been blocked for operational reasons by the DOD Enterprise-Level Protection System.
APPLICATION: qos-mission-critical-pan"
In some sense, traceroute does often give you some location information.
However, if you're talking about GPS or other geolocation information, even now if a machine supports IPv4 and/or IPv6 and isn't a phone, it likely doesn't know any geolocation information about itself. No provision for geolocation information was provided in the ICMP echo or echo response packets.
http://en.wikipedia.org/wiki/LOC_record
Took us an afternoon trying to find out their origin and make sure they weren't malicious but in the end they were just eating watts.
Later, when the project ended no one bothered to go through the whole formal process to get the machine out of the premises again. The machine was just left there connected to the network.
In the months after the project ended different departments moved into the building. Different people with different tasks. Yet the machine stayed connected. I guess no one had the courage to shut down a machine they didn't know about.
For a year or so I still used to check occasionally if I still could log into the machine. It disappeared eventually but unfortunately I don't remember how long exactly it lasted.
Having a perfect CMDB for hardware and following all procedures step by step could identify / prevent cases like that. But in reality, in normal environment it's going to happen.
Some of the data was so bad that we ended up having one guy who's job was to physically audit what hardware was really in the data centres, and then attempt to reconcile that with the CMDB data.
Next year I'm going to find that system on the 20th anniversary of that page.
I think these servers will outlive me.
http://www.theregister.co.uk/2001/04/12/missing_novell_serve...