UPSs, in my experience, quite often have or develop faults that defeat their purpose. We just threw one out that would cut power entirely during self-test, but only during self-test, not during an actual outage. I've seen their battery monitoring circuits fail such that they report a battery as good until power is cut and then die immediately. The other way also happens, reporting all batteries as bad even though they might be brand new, so you're never sure when to replace them.
I've come to the conclusion that if you are going to use a UPS, your equipment should always be plugged in to at least 2 UPSs, either via redundant power supplies or an ATS PDU.
> I live in a place where blackouts are very infrequent (perhaps a couple per year), but occasionally the mains drops out for only a second or two. I suspect these very short dropouts occur when substation switchgear operates, but have no way of being sure. Anyway, with a server running 24/7 I obviously wanted protection against any loss of the mains supply.
> So I thought I had covered all bases, and, indeed, the UPS proved useful on several occasions. I would quite often be on a work trip and receive an e-mail from the server informing me that mains power to the UPS had been lost, then another e-mail soon after informing me that mains power to the UPS had returned. Only once did the power cut last longer than the battery capacity, and the server was shutdown automatically.
It sounds like the UPS is doing exactly what he wants it to do. This is exactly why I bought a UPS. I would have power transients sometimes and rebooting my entire network (with NAS and servers) was annoying, and I didn't want it to go out when I was gone.
His main complaint is he bought a standby UPS where the battery failed, it turned off the UPS (which is understandably frustrating), and now replaces the battery every three years. Looking at the white paper referenced, it even says to go for Line Interactive for servers. I'm also curious how the person knew that power wasn't cut when he was away?
So other than buying the wrong type of UPS (which at face value looks to be a valid complain based on research)....why isn't a UPS as useful as they'd think?
>a standby UPS where the battery failed, it turned off the UPS
I assume this would be because the power is always flowing through the battery/inverter? Wouldn't that kill the battery a lot faster? Although it'd make the switch from mains to actual battery power basically a non-event from the output's perspective.
Now that I think about it, going 120/240->low voltage dc->120/240->12/5/3.3 is incredibly silly. IIRC there are 48 (or -48?) DC power supplies out there, mostly for data center use. Does anyone make an ATX version that you could hook up to a UPS that has a DC output?
Standby UPS topologies like the APC Back-UPS lines keep the battery out of the loop until a loss of power from the AC input is detected, at which point the inverter kicks on the the transfer switch takes the feed from there.
This is a design choice from APC, money is on the control electronics being powered off the battery meaning a dead or missing battery means the entire unit is non-functional. Making the unit resilient to this probably costs as much as a line-interactive unit, however, so you might as well just tell people to get a better unit than make a middle-ground solution.
I have a line-interactive APC and it most definitely is just a UPS-shaped brick without a battery. Might have changed in newer models, I wouldn't know.
My APC BackUPS Pro (supposedly line interactive) runs fine without battery (ran it with a tab from shipping still in the case), might depend on the setup. Generally line interactive makes little sense why it wouldn't work without battery since the power rail is shared.
Using a DC auxiliary power supply for backup is common in telecommunications equipment and by extension common in network appliances - "redundant power supply" for network equipment typically means a built-in AC power supply and then DC connections for an external aux power supply that provides battery, often called an RPS.
This isn't common for servers though which, if equipped for redundant power, have two AC power supplies. I think this just comes down to the higher current draw usually associated with servers, running even 48VDC power can become costly and impractical when you need to supply say a couple thousand watts per rack, so the extra space and thermal load of an AC power supply in each unit becomes preferable.
The downside is that while redundant power supply is usually a "standard" feature on switches and routers (that is, they always have the connection whether or not you buy an RPS is up to you), you need to specify and pay extra for dual AC power supplies up-front in servers, so you don't get as smooth as an upgrade path. That said in datacenter environments you often buy servers, racks, and power systems all at once so it's not as much of a concern.
There's a similar tradeoff that exists around bottom-of-rack UPS and central (building or area) UPS---bottom-of-rack UPS tends to be more expensive, higher maintenance, higher thermal load, etc for large installations, so usually large installations use one (or two for A+B power) large central units which may be hybrid between different technologies like flywheel and battery (and you could view the transfer to generators as a "third stage" of a central UPS system), but it also makes your cabling a little more complicated and you have a bit of an "eggs in one basket" situation. Central UPS generally have internal redundancy so their risk of failure is low, and often a bypass device where if a non-recoverable failure of the UPS is detected a contactor "shunts" the entire UPS removing it from the circuit so you don't lose power due to a UPS failure, but that doesn't necessarily save you when the HVAC pours water directly into the UPS control cabinet causing widespread failure of the control electronics... a situation that I have somehow seen twice. I bet on bigger installs you can get an external bypass with some sort of health monitoring though? If HP/Tandem taught is one thing it's that you can always through more redundancy into your very special basket.
As a person who has never bought a UPS, this behaviour of turning off power when the battery dies even though there is mains power is unintuitive, and I would have most likely hit the same issue. I see the post title as a PSA, in that if you happened to buy a standby UPS without knowing better, it might not be that useful if you very rarely have power interruptions but the battery keeps dying every couple of years.
Trust me.....having infrequent power outages like that are very very annoying. Everything turns off and I have to spend a non-trivial amount of time bringing my network back up (first router, then NAS (encrypted to I need to hand jam in the key), then servers that depend on NAS (also encrypted)). What eventualy got me to get a UPS was during a storm, it happened twice within an hour! This isn't even talking about if I were away, now my servers are down and I have to do it when I get home.
That UPS was well worth it to put a stop to that happening.
I ran into an extremely perplexing problem a few years ago where shutting down one server connected to a UPS would kill the entire network until the server was brought up again. The server was not providing any kind of networking at all, no DNS, DHCP, routing, etc.
After wasting a lot of time remotely trying to diagnose this, it was obvious what was happening once I got on site.
It turned out that once this server was shut down, the UPS didn't have enough power draw to power the other equipment attached to it, so a switch that was plugged into the same UPS would shut down as well. For some reason it had failed in such a way that it required a certain level of draw for it to function at all.
That reminds me of a "feature" in many of the consumer grade UPS units on the market now. I don't think it's enabled by default, but they do have an option to power down the UPS when the primary outlet is no longer seeing any power consumption.
Some UPSes have a special outlet on them that when it's not drawing power, it intentionally shuts off the other ports. It's meant for power saving, but I've seen more than one person get tripped up by this functionality.
> I'm also curious how the person knew that power wasn't cut when he was away?
Most homes have something with a mains-powered, non-battery-backed clock. You get home and the microwave or the oven is flashing 12:00 and you know there was a power cut.
> So other than buying the wrong type of UPS
This isn't a failure of 'type' of the UPS - it's a design fault deliberately left in by the device manufacturer for market segmentation.
It would be trivial for them to keep the power outputs on if the battery failed but the mains remained on. Leaving it out is a business decision, hobbling their home product so it doesn't steal market share from their small-business product.
Generally line interactive is less preferable since they provide less protection from surges than a double conversion setup. Big price difference though so line interactive is usually a reasonable compromise.
For a home setup you aren’t going to be spending the money a double or delta conversion unit costs, so line-interactive is the only sane choice.
You really don’t want line-interactive gear when dealing with larger loads either, as the transformer will just cause extra current draw during a voltage sag. This is why you don’t see (many of) them above 3-5KVA.
With all of the recent advances in Lithium-type battery tech (e.g. 18650) it seems like a market opportunity to create a new, modern, small, smart UPS.
These low-end APC dinosaurs still use lead acid, and in my experience (with a solid utility over the past decade) outages are just as likely to be caused by battery failure than some kind of actual power outage.
If anyone wants to build something better, happy to chat further.
Main advantage of lithium-ion over lead acid is lithium-ion is more energy dense so you can have a smaller size at a smaller weight. Important if you're building cars, but the price premium probably isn't worth it if you're building another office box at commodity pricing.
Edit: Since we're armchair-building for a predictable, high-uptime application, failure modes might have an interesting advantage though. Any battery engineers know if lithium ion has more predictable or measurable failure modes? Yes, we've all seen the exploding hoverboards... I'm assuming theoretically well-built electronics.
It could possibly target the "Apple-esque" product category by using a smaller physical footprint to emphasize sleek looks, perhaps with a small sharp readout. The small size could be leveraged further by only attempting to provide power for a short amount of time, say 10 minutes, emphasizing and requiring USB connectivity to shutdown the protected computer at minute 9.
Lithium batteries have a much longer lifespan than lead acid, which would eliminate the need to replace UPS batteries every few years. They are also more power dense, so if the goal is just to keep a machine running long enough for a clean shutdown, you can get away with a smaller (energy and size) battery for the same load.
There is a reason we don't see lihion car batteries, or fork lift batteries as a default. (Yes, Tesla's are li-ion, but that's a scale where the weight savings begins to matter. Iirc those battery packs can't deliver peak amperage quickly or many times in their life, e.g. ludicrous mode.)
Additionally, li-ion has non-neglegable disposal costs, so I don't think it's fair to count that solely again lead acid batteries.
I don't understand how lead acid has an advantage in recharge cycles? My lithium batteries can be discharged and recharged over 1000 times before they wear out, and they hold a charge for ages. My UPS with lead acid batteries, however, is practically useless after a single full discharge, and the batteries have to be replaced practically every year.
Portable power packs are crazy cheap these days. You can get a 20,000mAh pack for $45 quite easily. They are almost always Lithium Polymer based, and are very small and light. I have had one sitting in my bag for almost a year now and it still has most of its charge.
What I don't understand is why it would be so hard to just scale those power packs up a little bit and slap a normal 110 volt A/C plug on them? Wouldn't that essentially be a super efficient, light weight UPS?
If a power outage is long enough for the battery to go flat, I need to power on things gradually, or my UPS will act as if overloaded. The stuff I've plugged in barely amounts to 280W. It's a home lab, so no much harm done, but also, so much for unattended recovery from failure.
It's good for brief outages caused by thunderstorms, though, so still better than nothing.
The explanation of the failure was very strange to me. I know there are three main UPS types - offline, line interactive, and online. Offline types have the battery disconnected until there is a power outage and only use it when needed. Line interactive and online use the battery more often to absorb power spikes, brownouts, and other anomalies. These two types also use the battery more and need replacements more often than offline UPS.
I have been using the same offline UPS and battery for the last 7 years and it is still working fine - a few days ago it handled a 15 min power outage with 50% of the battery.
Can someone explain why offline UPS can fail if the battery dies? Is it by design or a manufacturer specific issue (e.g. only APC brand ones do this)?
because he gets emails from the unit when it happens.
I had a script in a raspberry pooling my UPS each minute asking if the unit was running on AC or battery, and dropping that info into a file when it changed. I know the software/daemon can also do something like that.
Yes, but in this case he wouldn't have gotten an email since the UPS failed right when the power went out.
I actually had the exact same thing happen to me a few months ago. A power flicker so brief that it didn't even reset any of my clocks, but it caused one of my cheapo standby UPSes to give out.
Theres a few more types than that. But thats the gist of it.
"online" is generally a double conversion UPS.
These are generally the best for clean power to the electronics. Since its always supplied by the battery. Its also often one of the hardest on the battery.
Line-interactive are probably most common. They are pretty solid, but since there is a delay for the cutover, surges and really dirty spikes can make it through to the equipment. So say a lightning strike or REALLY bad surge on an overloaded generator can get through, whereas on a Double conversion it may just trip the fuse or breaker. Also running a whole home or standby generator on these line interactive can make them trip constantly since the generators often dont
A) run at 60Hz (ie: mine runs closer to 63Hz)
B) run with a pure sine wave
You can get inverter generators to help with this. But thats a cost too. And otherwise the solution is to "de-tune" the UPS sensitivity.
I de-tune this on mine. Its been fine, running on gennys during the rainy months for hours and every know and then...days at a time.
I have had to swap out power supplies more often..But even then, usually thats after like...5+ years of runtime.
I was just pricing a UPS for home office use this week as we have frequent outages where I am but became a little stuck when I realized that to get a UPS that won't barf when my generators come online moments after an outage, I'd need a double-conversion UPS, which is $$$$. A little stuck for a reasonable home office option given this wrinkle.
I mean. Im using line interactive ones and have for 4-5 years.
I have to de-tune their sensitivity but they work.
But it does stress the CPU power supply caps a little more and shorten their life. But the caps on the board are fine, and havent had an equipment failure outside of a PSU...And ive run literal weeks on genny power.
But that stress is more due to Hz being at the threshold and artificial stepped sine wave i think. Voltage fluctuations when load kicks up certainly doesnt help.
You can silence the alarm by pulling the battery connection. Obviously, that's inconvenient if you want the UPS to be useful again as soon as the power comes back.
I personally like the white noise from electronics, so I can't help you there.
There is a way to turn off the beeps on the APC UPS line (at least the ES 700 I have), but it involves installing software that comes with it onto a Windows machine, connecting to the UPS via USB and changing the settings.
It's enough of a pain to do that rather than configure all 3 UPS devices in my house, I've only ever gotten around to doing it on one.
Can't say I've noticed the electrical hum, except during a blackout. But I don't have mine in a bedroom.
The hum is caused by cheap or warn lamination on the transformer. I recently "fixed" an older APC by opening it, removing the large transformer, coating the fins in nail polish, and adding rubber washers to the mounting screws.
> The loud can't turn it off beeps when power is out.
That depends on the UPS, it seems.
I have a pair of Back-UPS 1500s with the external battery packs. They have a button on the front-panel for disabling/enabling the audible alarm (just hold it down for 2s, IIRC).
When the power goes out on an APC UPS, just press and hold the "On" button until it chirps. Alarms are silenced.
Now imagine the sound at the APC factory, when the power goes out and thousands of UPSes on the charging racks start beeping in sync...! (this is not a hypothetical)
If you want a poors man home server just repurpose old laptops. I do with a bunch of them and it's fine. I know laptops are not for that, but they wouldn't have any other use anyway, and it's like a server with an UPS built in.
Most tasks are running scrapers or data wrangling so I don't have to keep my desktop running.
I've been doing this for maybe close to two years, and for now it works. I wouldn't recommend for VMs though.
Also, your router needs an UPS too but it will be fine with a cheap one.
Why not? I have a laptop with a 4th gen i5 that has VT-x. I've upgraded it to 16 GB RAM and an SSD. For now, it's a Spotify connect player but I'm contemplating running a couple VMs on it too, like pi-hole and whatnot.
Sure, that’s still reasonable, but it’s not young.
I’m also using similar class hardware for my VM server at home. Works well enough for home workloads, and for everything else there’s the cloud on demand.
I have an thinkpad x220 with some core-duo on it and 4GB Ram, can't remember, and it really was struggling with VMs. Didn't really cared much to check the specs. Just tried some debian VMs and it was less than satisfying.
I've been running a 2008 MacBook Pro as a server for 10 years. I've replaced the battery once, about 2 years ago, and one of the fans about six months ago. Other than that, it's been on the whole time.
What OS do/did you run on it? Thinking about repurposing an old 2011 Mac Mini as a server, it should still be more than capable, but since it stopped getting newer macOS versions a few years back it wouldn't get security updates, that kinda worries me.
Just checked, it's actually a mid-2009 MBP and its running 10.11.6 - it's not had a security update since 2018, which yeah, that is a little worrying. Might wipe it and install Ubuntu if there's such a thing as Firewire800 drivers for it.
I run a Raspberry Pi 4b, recently upgraded from a 3b+, and I believe that a lot of personal computing could (and maybe should) be run on them. They're quiet, energy efficient, and you could probably run them on a battery if you needed it.
I wouldn't run VMs on it, but I do run docker on it, and that works just fine.
what's uptime like on an Rpi? I want to start working on a home website and have an old rpi laying around. But I've read that their reliability isn't great since they use SD cards. Any advice?
I don't have numbers, stats handy but in my couple of years experience with Rpis - both regulars and the smaller Ws - the SD cards fail well before the Rpis do.
What I've done with my RPi 4 is attach an external SSD to it and mount it to /home and /var. This reduces writes to the SD card dramatically. Lengthening the SD card's life. I have serious uptime on it, almost a year (since I got an RPi 4 basically). It only shutdown with power failures (only one, which was entirely my own fault) and system maintenance (OS upgrades and other tinkering).
I've also imaged a version of that SD card, so it would be easy to just swap out and probably get things running again without much hassle.
I'm sure there are more fancy solutions with overlayfs, making the SD card read-only except for upgrades. Though I couldn't find a nice resource on it at the time.
My rPi3, running Arch, has an up time of close to 3 years on the same SD card. It runs various Docker containers including 1 for MiniFlux. It's mostly idle but when I need it, it's been reliably there for me. Maybe, I've been lucky.
My Home Assistant is running on a Raspberry Pi 3B+. It has been running on the same SD card for two years now, with a large sqlite database and with debug logging enabled for Z-Wave so a fair amount of writing to the card.
It has only been down when the power has been out, and so far no issues with the card. It is however an A1 or A2 class card (I forget which), so supposedly design with applications in mind.
It should also be noted that a lot of issues with the Pi is due to poor USB cables / power supplies. I had another Pi that kept crashing every few days, until I measured the USB "charger" cable I was using for it. Turns out it had a resistance of almost exactly 1 Ohm. So if the Pi drew say 1A, that would be 1V loss in the cable... After I swapped cables it has been rock solid. A key point here is that a lot of SD cards do not like losing power while being written to. Avoid that and a quality SD card should have a long life.
Just because I already have it and it's low-powered. I prefer starting projects out with 0-cost methods, and I already have an rpi laying around somewhere
That is a good point. However one of my goals/constraints is that I want my home site to be able to connect to other personal devices, e.g. a NAS (another WIP project). My ideal vision is a simple site running on my network, backed up on some kind of cloud storage.
Any suggestions? Though I'm a software engineer, I've worked mostly in app dev rather than infrastructure setup. This project is a way for me to learn more about the latter. The rpi is the easiest and cheapest option, but I have wondered if it wouldn't make more sense to use AWS or something like that. I wouldn't mind paying a few bucks a month, but more than that and I'd rather wait until I've learned a bit and know what my specific needs are. I've also considered an actual computer, but figure it's a waste of power when my initial work will probably just be figuring out the server config, HTML pages, etc.
>That is a good point. However one of my goals/constraints is that I want my home site to be able to connect to other personal devices, e.g. a NAS (another WIP project). My ideal vision is a simple site running on my network, backed up on some kind of cloud storage.
You could host everything on your home network but there are a couple of things you should probably know before you start: 1) Some ISP do not give you static IP, meaning its extra work
2) Security - Ideally your home network has some kind of segregation, you keep your trusted devices on one, IoT or not patchable on another for example, with firewall blocking connections from dirty network to clean one. Having externally facing website potentially having direct access to personal devices and services might be an issue from security point of view.
You can sign up on AWS/Azure or any other cloud provider for free and get free credits, which might be enough for you to learn. Feel free to ask further questions if you need to
Just dont have heavy writes on sdcard. I use external hdd for it.
I have a running rpi for more than half a year. No issues whatsoever. But again - I have a very stable electricity and I dont remember last time it was cut not due to planned maintenance.
> I think the pi is not particularly power efficient.
While there might be efficiency gains available, RPIs use a very small amount of power compared to a laptop. Idling a RPI4 uses 2.8 watts[0], which not only rates favorably compared to most laptops, but is far below what my charging cell phone uses. Maxed out my RPI can only hit 15W, since that's the maximum power that my official USB C adapter can output.
Now that being said, it does depend on what kind of load you're expecting to handle. If you're constantly maxing out a RPI, it is probably more energy efficient to purchase a larger server than to just keep adding RPIs. But if you're staying well below the theoretical max of a RPI, it'll consume far less than a used laptop.
> I think the pi would benefit from a more robust filesytem layout such as an overlay filesystem to allow continuous writes to /var to go to ramdisk.
This is one of the biggest drawbacks of the RPI; microSD cards cannot handle a ton of writes without frying. I've settled on using log2ram to ensure that logs are only periodically flushed from RAM to the SD card, to extend the lifespan of my cards.
Watts per low traffic website it's very efficient. CPU is generally not pegged at 100%.
For watts per CPU cycle not as much, I agree. Say it takes 10W at full power (I've not measured). That makes it 1/10th of a normal CPU. But it's like 30-40 times slower.
Traders still seem to be the most gullible when it comes to buying unnecessary hardware.
They really think their prosumer workstation is going to give them an advantage.
I guess thats one silver lining about Robinhood’s proliferation, many people know its good enough. Gullible traders still misread options settlement UI but seems there is a selective evolution at play to make that less common too.
Given how unoptimized frontends for crypto exchanges tend to be and how rapidly the trading volume could spike in these markets, actually it might make a difference.
Been running a 2010 MBP for the last few months hosting bitwarden, a samba file share, a DNS server, and a ton of other stuff. It's been pretty great so far, so much so that I dug out another old laptop and set it up as a pi-hole for a friend.
Have you any suggestions for increasing the amount of storage one could address? I can't find any Thunderbolt 2 or FireWire enclosures so I'm sadly limited to the two USB3.0 ports. Right know my plan is to resort to putting a high-capacity 2.5" hdd inside the MBP and adding a usb hdd in addition to the one I already have. But that doesn't strike me as ideal for some reason.
Well, used to be. As I understand it newer laptops are more and more often using pouch style batteries to keep things thinner by using any available space.
They're just general computing devices. With the right software (and/or some hardware) you can turn them into whatever you want. CCTV, web server, door/gate intercom, electric meter reader/submitter, NAS (maybe with multiple drives using a powered USB hub), Wifi AP, media center, desktop computer, etc
Yeah I just repurposed a 2015 MBP as a desktop now that it is painfully heavier than my 2018 12" Macbook. I just found out that it could do 4K60 through the display port, and as a desktop machine it's great.
Then the next level of slipping down the chain is the headless server or timelapse photo taker. :)
I was considering this but then I stumbled across some ~7 year old micro towers. Those little thin client boxes that were popular for a short time as a thing that can connect a keyboard, mouse and monitor to a corporate network.
Refurbished with a Windows license (if I ever want to use it) was less than a surplus laptop. Power usage seems acceptable but I'll compare with a Kill-A-Watt. I have one on the way and if it works out I'll grab a few more.
I do like the idea of a surplus laptop cluster, mostly because of the built in UPS factor but there's a lot of wasted hardware there, such as the monitor and keyboard. But I suppose that's kind of a feature too.
I have shower thoughts of some kind of custom bladecenter made of old laptops but that's a lot of work for probably zero benefit.
To me the main problem would be the noise. I don't have a dedicated server room and even if me and my wife's bedroom + living room is quite big it'd still be annoying during the night. But I am quite fond of the idea to reuse old laptops regardless.
Can you post a link to those 7 year old micro towers, please?
Yeah noise is why I didn’t go with surplus servers, and the power requirements. Proper servers are actually pretty cheap, less than these tiny towers even. But the power and noise is just too much. Plus less horsepower is actually more interesting to me so my workloads have a hope of hitting scaling limitations.
They’re available in a bunch of configurations and on multiple sites so you can look around for a good deal that fits your needs. I found one with an SSD for additional power savings.
In regards of power efficiency I'd still be drooling over one of the Xeon D-1600 configurations but haven't found one that's not in a rack form factor yet.
As a router, it's quiet in a desktop environment, unless the CPU is fully loaded. For a bedroom, silent is better than quiet, which could justify investment in a fanless case for mini-ITX. These are not cheap, e.g. Streacom or Akasa.
Broadwell NUCs are available on eBay and Newegg has a matching Akasa Plato X case for $100. With an SSD, that has no moving parts. But only dual core, single NIC.
In most cases the beefiest air cooler + the lowest tolerable fan curve (meaning no overheating) will do the job just fine, especially if the CPU TDP is between 35-65W. Alternatively you can also limit the CPU power usage by disabling turbo or doing other tricks that forces it to run slower, but more efficiently.
Woah these are really cool. $$$ for what amounts to bling in my case but I might still be interested in something like this. The NUCs are great but I don't see many of them on the second hand market. At least not as many as these thin clients.
There are advantages to this, but do note that earlier core i* chips had earlier versions of QuickSync that work, it just wasn't as good as new versions (5th gen and newer).
Plex can use nVidia cards for hardware acceleration for encoding.
Different gear, but you’re starting to see HP t630 thin clients on the market for <$100. They are silent and have standard m2 interfaces for storage and sometimes come with a bunch of memory.
Noise can be lowered cheaply on adapted laptops with a little work. Re-paste, cut out the fan grill plastic, keep the HSF assembly clean and fluff-free, run the fans slower. Keep the unit elevated on feet.
Replace the fan with a Noctua-type quiet model bolted on. If the HSF needs that airflow directed, hack a simple shroud (or 3D print a replacement.)
Another upside is laptop chips suck less power but still perform ok in most use cases, especially without a graphical load. The monitor and keyboard allow easy interfacing if needed plus ports are often convenient too. even just eSATA for more storage.
I run a uSFF HP box for my hypervisor and a low power J5005 core in my NAS and am more than happy with the performance let alone minimal power usage. Sure the J5005 system takes ages to update requiring a backup DC/PiHole setup on the HyperVisor but it just works at low cost.
This has served me well for a year ($80 Dell laptop w/ an i5 from Craigslist) but I did just splurge on a Synology NAS and an APC UPS because having multiple external hard drives plugged into the laptop was quite dicey. The laptop is still a Plex server but now I have peace of mind for the data.
I'm curious, with shared servers at Digital Ocean available for $5 or $10 per month, why run a home server anymore? What is the value to having it local?
I've been considering this calculation for a year or so now. I spend an average of $20/mo on small VMs for testing (including the occasional $1/hr VMs for a couple of hours for heavy loads).
I thought I should replace my 2012 Mac Mini (HTPC) with a modern Hackintosh and 6c/12t or 8c/16t and do my testing workloads on that (or even just use my current 2c/4t for small workloads). But it's just so easy to spin up DO droplets within seconds and do so programmatically. Plus you can snapshot and create droplets from those, and turn on backups if you need to.
Like the author, I rarely see power interruptions that last more than a second or two (unless there's a major snowstorm or something).
It's kind of annoying and I'd love to see a solution that isn't just "get a battery based UPS" -- maybe something capacitor based, enough for say 15-30 seconds?
I guess I could get a flywheel [0], but I don't think it'd fit in my basement.
Back in the late 90s I had a PC motherboard that had a capacitor for exactly this. I could (if i was quick) change which outlet the pc was plugged into without loss of uptime
I also remember a green pc set up which used one of the low power x86 clones (Via?) in the 90s which used the bios battery double as a mini ups similar to that.
At the reasonably sized datacentre that I ran we had a voltage conditioner that also had "sag protection". It was an HPC shop and we decided that it wasn't worth the money to put the compute on UPS (just storage and core servers but the Top 500 machines ran off mains) but we still had 10s of millions of dollars worth of hardware that a lightning strike etc could fry.
The conditioner provided a constant voltage to the servers and could correct "sags" where the voltage would drop, and it could keep things going for a few seconds if the power went off completely. As you say this took our total outages down from probably 10 a year to maybe 1 or 2 - most outages are quite short. Although looking at the graphs on this thing, there were pretty frequent surges and sags coming from the street, like when you see light bulbs flicker in your house.
One other thing we noticed was that if there was a short interruption, all of the HP servers could survive for a second but the Dells turned off right away. Different power supplies store up some amount of power locally.
Looks like it's capacitor-based, removes the need for an inverter, and even has a clever way of communicating to the host, by taking over the power switch connector.
Unfortunately, I can't seem to find a price or where to buy. Would love to see more momentum around products like this, and marketing directed at home systems.
EDIT:
Thought of one more system that uses capacitors to limp to a clean shutdown--the Unifi Cloud Key Gen2.
Again though, the power solution isn't packaged for consumer use.
Though, maybe it'd be amusing to try to hack together a NAS out of Cloud Key Gen2 Pluses... there's a hard drive bay in each one ;D
One thing I always wonder about these home server is this. When you are leaving house for a long vacation, is it really safe to leave some electrical appliances working? Aren’t people afraid of fire hazard. What if there is a short circuit while you are away. A smelling short circuit would definitely burn something if left unattended for few days.
For once there was a high voltage fault in my neighbourhood, destroying few lights, all mobile chargers plugged in and microwave left in standby. Had I not been home, and fault lasting more than 5min, things might have gone really bad.
Now on, Whenever I leave my home for long duration, I just empty the Refrigerator and switch the house mains supply off before loving the house.
I can answer this as I’m currently on a two-week holiday and I have a small homelab running at home.
First and foremost: fire alarms. Buy them, and use plenty of them.
Secondly: let your neighbors know you’re on vacation. This implies a certain amount of trust with your neighbors, but it’s good to be on good terms with them for more reasons than just this.
Thirdly: I get notifications if a server goes down or a fire alarm is triggered.
With all this in place, I don’t worry at all. I may be somewhat ignorant or oblivious to certain risks, but in general I trust the safety net I just described.
I find it surprising that service panels don’t have more extensive protection features. I would like my main breaker to trip under any of these conditions:
- Sustained overvoltage. Any overvoltage lasting too long to be handled by a surge protector should trip the main breaker.
- Sustained undervoltage. Some devices don’t like this.
- Phase error. A two phase main should detect incorrect phase-to-neutral voltages and incorrect phase-to-phase voltages. A three phase main should also detect incorrect phase sequence. This should also catch some cases of a disconnected neutral.
- Excessive neutral to ground current. In the US, for reasons I personally strongly dislike, every “service” has a connection from ground to neutral, usually in the meter box or the main panel. Various errors can cause substantial current to flow through this connection and can be dangerous. For example, if a house and the utility both have excellent grounds but the house’s neutral feed breaks, all the voltages can be close to correct and the house’s neutral current will return to the utility through ground. This is dangerous and will not be detected by common equipment. (A whole-house residual current detector would satisfy this, too. I think these are used in Europe.)
One thing I always wonder about these home server is this. When you are leaving house for a long vacation, is it really safe to leave some electrical appliances working?
I used to feel this way, but then I realized that the electronic devices don't know I'm away on vacation, or when I've returned, and they're just as likely to start a fire when I'm down the street getting coffee as on the other side of the planet. And when I return, my presence doesn't magically reset some catastrophe timer and everything's all better because I touched "base."
That said, if I'm gone from home long enough that I turn off (or down) the refrigerator, then I'll unplug everything. So I guess I'm still a bit paranoid.
There are failure conditions that are not immediately catastrophic. The reasoning being that if you come home after a work day (or even better, after coming back from your coffee down the street) and see that a device malfunctioned you will attend to it. As opposed to leaving it plugged in and turned on for weeks while you are not there.
A lot of devices will smoke before starting a fire, sometimes for quite a while until temps increase sufficiently for ignition. You would smell the smoke and figure out the culprit.
Turning off everything is still worthwhile while you are away. This is just risk management. It is a small risk (maybe you have faulty appliances or house wiring? maybe you live in an earthquake or tornado area? etc) but the mitigation is also not very inconvenient.
That said, if you can have a smoke detector pinging you (or a similar device - Amazon's cameras have a microphone that can alert you to alarms) it would be better. As you point out, stuff could happen while you are temporarily outside the home.
Short of being home and reacting immediately with a fire extinguisher when the smoke alarm is triggered I certainly agree with you that there's little that being away changes w/ respect to the electrical fire risk profile and potential for loss. Whether you're gone for a week or gone for 15 minutes makes little difference in responding to a fire.
Responding to a water-related disaster is a different story, though, and worth bringing up.
You might want to consider turning off your water at the main if you're going to be away from home for an extended period. I was fortunate in that the only water-related failure I've had personally was a ruptured water heater leaking into a floor drain of a 1st floor utility room. I know others who haven't been so lucky (having clothes washer hoses rupture, fixtures fail, etc) and have had water running inside their home for an extended period (several days, in one case). Until someone notices the local municipality reading excessive usage, a neighbor noticing water running out a patio door, etc) the damage continues and, to add insult to injury, you're also getting charged for the water / sewer usage.
Before the pandemic I spent ~1/3 of my time away from home anyway (work during the week, either out camping or otherwise out and about for at least 1 in 3 weekends). I certainly didn't unplug everything every time I left for work. I don't really see a significant difference in risk profile for a week long vacation.
Electrical fires inside stuff that isn't the cheapest straight-from-china import shit are incredibly rare. Most fires causes are A) very old wiring that has never been attended to but loaded to the max B) people messing with mains wiring and causing a fire (e.g. bridging thermal cut off switches in cable drums) and the occasional person drilling into a cable in the wall and hitting it just-so that PE is untouched but L and N are getting just cozy enough to start a fire without tripping the line breaker.
* Back-UPS: inverter offline, expect a 15-20ms power transfer
* Smart-UPS: interactive, inverter is always online 2-5ms trasnfter
* Symmetra and above: inverter online, voltage conditioned, 0ms transfer
Most desktops will work just fine with the Back-UPS line. The processor is practically sleeping. Workstations should be on a Smart-UPS because the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
I have the next step up from this, the 1500 VA. I assumed the UPS under my desk was the Smart-UPS series but I can see it is the Back-UPS Pro. I haven't been in the office in months!
I had 3 hard drives (likely) fail because of brownouts regularly occurring at a residence when high-amperage devices kicked on (power tools, air con, hairdryers, etc). The line would drop into the 80-90v range (120vac) which wasn't enough to turn anything off, but you could notice the lights "flicker." After a voltage conditioning UPS was added, haven't had a hdd failure
For situations where a UPS is too expensive or impractical you can also buy pure voltage regulators that use some kind of autotransformer, like the APC Line-R. I worked for an institution for a while that had every laser printer on one, there had been some bad experiences with losing laser printers to overvoltage incidents but they generally draw too much current to go on a UPS. I've also seen them used for A/V equipment.
Hold-up time is one of these things (besides long full-load life) that separates cream from crop in power supplies. High-grade ATX power supplies don't just promise 25+ ms hold up time, but actually deliver.
Some better review outlets test for that, since you can't rely on PSUs actually managing the spec-required 17 ms (I think that's the number, something in that ballpark)
Looking at the schematics of some old models of BackUPS I found online the output relay needs to be powered in order to supply output from mains. Combine this with slightly funky internal power topology where +12V for internal logic comes mostly directly from the battery and it is pretty clear why battery with shorted cell(s) causes the thing to completely shutdown (probably even due to thermal shutdown of the LM317 in charging circuitry).
> Workstations should be on a Smart-UPS because the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
Also make sure it's rated for the load. My UPS went "lol nope" when I lost power while I was gaming, because I had forgotten about it when I went from single mid-range GPU to SLI top-end GPU...
Lots of users have desktops, monitors, chargers, and even heaters plugged into our UPSes. To them it's just a chunky power strip. They're lucky if they can squeak out more than 5 minutes on battery.
It's more complicated than that.
There are Smart-UPS online. Thats double conversion also 0ms transfer
Some of the "newer" online models will have a green mode, which although double conversion green mode introduces a transfer time (~2ms, basically relay lift time).
Backups are also square wave, or stepped sine
Smart are Sinewave output.
This is specifically APC.
Also remember there are tare losses (copper / magnetizing and conversions), so a UPS just sitting there can use up to 30W.
Back-UPS stepped sine wave has about 3 square steps above or below zero, so it's a very rough approximation of a sine wave. Adequate for many kinds of equipment, but ugly on a scope.
There is, or used to be, a pin on the Smart-UPS line that you could tie high (or low?), to tell the device to operate in "Online" mode (inverter always active, line power continuously tending the battery). APC also sold it preconfigured that way as a separate, more expensive, product. :)
There's also a trick to jump-starting an unplugged-but-charged Smart-UPS. I've used this for several hours of remote power in inconvenient locations.
Quality devices, but test them regularly. The switch from idle to high-draw can be violent, and it takes out weak components. Similar to tungsten lightbulb filaments.
Oh, I know very well it's complicated. But that is the 500ft explanation I give to the beancounters, "why are we using these expensive UPSes when we can get them for $100 from Office Depot."
I'm most familiar with the APC line because that is what I use.
> the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
I'm not sure I buy that. The motherboard/GPU backplane VRMs alone should be able to buffer 20ms on these devices. Would love to see a test of this kind of thing.
> While I was away from home on a long work trip, suddenly I could no longer connect to my server and I had not received an e-mail from the server informing me of any problem. Luckily it was near the end of my trip so I was not too inconvenienced. When I arrived home I found that the UPS was sounding an alarm and was not supplying power to the server even though there was mains supply to the UPS. It transpired that the UPS battery had suddenly died without warning and could no longer hold a charge, and this had happened while there was mains supply to the UPS, i.e. there had not been a power cut while I was away. Fortunately there was no loss of data on the server; I was able to run fsck during boot-up.
And this is why professional servers have two, redundant, power supplies.
If you had two power supplies, the UPS would fail, but you'd continue to function off of mains voltage.
UPS can fail. Mains can fail. Two power supplies hooked up two both powering your server redundantly means you'll only fail if both fail simultaneously.
Also, surge protectors wear out and should be replaced before they lose the ability to protect.
This will probably occur at a different time than when the UPS wears out, so it could be beneficial to keep them separate so you can replace the surge protector only.
You don't need an UPS to protect against surges. In fact, UPSs aren't surge protection devices; they may filter a few of those, but they should be protected too.
That's not really a proper way to deal with it. Proper UPS for a home server or any standalone server or appliance has to hook up either after ATX power supply in parallel and provide all the ATX voltages or hook up directly into a power supply to save a bit on extra converters. This way UPS failure won't affect anything and such UPS will be much more efficient.
It's just that consumer home UPSes are almost scam level products designed to profit from naive people. They are both not improving things for an average user and killing batteries after like a year of operation (lead acid batteries can last for like a decade when not subjected to those UPS chargers).
My cheapo UPS certainly improved things for me when I lived in an apartment where the entire unit minus kitchen appliances was wired to a single 15A circuit. I tripped the breaker semi-regularly by forgetting to turn off the AC before using the vacuum cleaner or toaster oven or whatever.
I live in a place where blackouts are very infrequent (perhaps a couple per year)
Ouch. Does anyone know where the author lives that a couple of times a year is considered "infrequent?"
Decades ago I lived in a rural area of America where we'd have power outages two or three times a year because of storms. But I've lived in more than a dozen cities since then, and the power has only gone out twice. Once about a decade ago during a tornado, and then again last year when the power company replaced a transformer across the street.
I ask because it's my impression that electric service has gotten much more reliable over the years. There are still access issues where I live now (believe it or not, thousands of Americans don't have access to the electric grid at home), but reliability seems improved. Am I living in a bubble when it comes to electricity?
I've had at least a hundred times more power outages due to protection devices tripping (usually true positives) than the grid going down, so spending 20 minutes to figure out which socket would be the best choice to hook all the communications stuff up to is a much better investment than buying an UPS. FWIW, I had more downtime with the UPS than through actual grid outages because the UPS didn't like a battery hotswap and powered off, or because the batteries started to gas off hydrogen sulfide, which required disconnecting them immediately, but the UPS then powered down (without a battery) and couldn't start without one, either, so it required rewiring a little bit to get going again.
Edit: My UPS is a line-interactive rack-mount APC unit, which as I understand things don't actually contain a power supply, so everything runs from the battery voltage and is supported by the charger in mains operation, but without a battery inserted the charger doesn't work, because its control circuit is powered by the battery.
I live in florida. And in an area where most power is overhead.
Storms during the rainy season usually means i experience cuts. Usually 1-2 times a month for 3-8 hours ea from May-Nov
Hurricanes are usually expected to be 3+ days at minimum of outage. Most recent ones were closer to 14 days.
I have a couple gennys, and interlock kits with 30a hookups on my house during these times. I can run 1 genny on the whole house, or even 2 since i have a sub-panel with another interlock kit.
The only thing i CANT run is my AC. But we have 2-3 window units/portables to get by.
FWIW running generators for days at a time is NOT cheap. on my setup its about 20-30 bucks a day in fuel alone.
Even with my roof (which is east/west facing) the cost to just install panels would be 20-30k. And my city has a 1:1 buy back...And even assuming i sell them back power it would take a decade to recoup the costs...and by then the panels would be less efficient. And this is with buying 95% panels.
but then during an outage....
So you need something to store it... Short of rednecking a series of deep cycle batteries the Tesla Powerwalls are really the only big option.
And those arent cheap either. To run my house for 2-3 days would be in another 10-30k
I can do all of that with 2x7kw gennys and some ancillaries. With a 14-17kw i could probably run my AC (which have hard start caps on them) Even at 30 bucks a day in gas im still well south of even 10k.
Next step up would probably be a active standby generator with transfer switch and giant Propane tank to fuel it. That would be about 10-15k once permitting and all is done. And that adds complexity and cost. But it would mean i dont have to manually flip breakers and cutover. (as it is, my computers are just set to talk to a NUT server, and shutdown in the event i dont cutover within 10 minutes.)
And during storms supply chains are strained. Its hard to source diesel and LP and sometimes even petrol (in fact i warned a previous company of this, and we almost had to shutdown due to fuel levels). And you have to maintain a contract with LP providers and usually rent a tank from them.
With petrol and my little standby units i can source it myself (3-4 am is the best time during runs on gas), even siphon from one of our cars.
I may spring for a full kit standby one day. But for me, it costs about 2k in generators. Another 1k to have an electrician install the interlock kits. And i can service/manage the gennys myself. swap them out while i do maintenance during extended runs etc.
They are panels with slight manufacturing defects that you can buy and wont have a full lifetime or near the normal efficiency. So are less expensive as well.
They used to sell them last time i was looking into it, which was about 3 years ago.
Lived in Florida for about two years. We’d lose power due to the neighborhood transformer exploding a couple of times a year. That must have gotten expensive real quick.
The quality of the residential wiring in the original half of the house was not friendly to my hard disks. I lost four in rapid succession. Got a newer UPS, line interactive, had better luck after that until we moved out of state.
(That month of disk meltdown made me love ZFS. Didn’t lose any data.)
| FWIW running generators for days at a time is NOT cheap. on my setup its about 20-30 bucks a day in fuel alone.
What type/size of generator are you running? Might be worth taking a look at an inverter unit next time; they're much quieter and sip gas. Can't stress the quiet part enough - you can hold a conversation a few feet away easily.
I have looked at inverter generators. But they seem harder to find in larger wattages/capacities. I dont know that i trust Champion engines for long runtimes (days)
I currently have a 7kw (8750 surge) and a 6kw unit. they are dewalt gennys with Honda engines. I really only would trust a B&S, Honda or Kohler engine i feel like.
I think my next purchase will either be a bigger inverter or just a 14-17kw contractor genny and switch to double conversion Battery Backups.
Not quite sure. But what i have does work too. Just that my current generator is approaching a decent number of hours (400-500).
Floridian here as well. Our lines are buried, but the feed is above ground so if a tree takes that out, a couple thousand or so of us have no power for hours. I've thought about a generator, but I can usually keep working on battery (laptop and phone) so, I've not yes seriously considered it.
It is highly variable depending on the location. I know of two electrical grids in the Bible Belt, less than 10 miles apart. One has a power outage every year or two. The other, 10x or more per year is not uncommon.
I've lived in many different states and cities, and "it varies" is about the best you can say. Areas I would expect to be relatively problem free, end up being some of the worst offenders (looking at you, El Paso County, Colorado).
Few times a week in summer and twice or so weekly in winters here (India). It's not that uncommon in developing countries to have outages. Many of them are also arbitrary (not related to infrastructure failure) but saving cost, political reasons, social events, etc.
I have a UPS mostly because my laser printer sometimes trips the stupid arc fault circuit breaker[0] that gets false alarms incredibly easily. Since my apartment is new construction it has AFCI on every circuit. The TV also used to trip it and maintenance replaced that circuit’s breaker and it’s stopped, and I’d probably get the other replaced too if not for COVID requiring all our maintenance requests to be emergency-only.
AFCI’s are such a horrific scam. Imagine spending thousands of dollars to fill a breaker box with less reliable breakers.
I had AFCI trip on my fridge after renovating my kitchen while we were away for a weekend and ruined the brand new floors when the ice box melted. So infuriating.
I’ve since pulled the AFCI’s out of all mission critical circuits and sold them on eBay.
Just an absolutely disgusting example of regulatory capture.
Germany here. I'd say from personal experience maybe once decade. That includes living in a village of ~3000 for eight years. One time I remember was caused by an excavator where it lasted several hours, the other instances were rather short.
I guess not having earthquakes, floods or hurricanes helps.
The other items at least in the US midwest would be ice storms (ice coating power lines making them heavy and branches getting too coated and falling on power lines) and thunderstorms (50+ mph wind coming through bringing down tree branches on to powerlines) or lightning strikes hitting power transformers.
It's a lot more common in Europe to have buried lines for anything after the high-voltage national backbone, so the grid is much less susceptible to weather.
Where I lived in Sweden, 30% of the network was already underground, but after a large storm in 2005 caused long-lasting power outages, they invested heavily and are now up to 70% underground.
| Am I living in a bubble when it comes to electricity?
Your electricity service may not be as reliable as you think - some utility power can be quite "dirty" with high levels of distortion. There's a lot of infrastructure between you and the power company that can cause problems. A good UPS will not only protect against outages, but will filter/condition utility power.
I live in Chicago, in an older apartment building. I haven't had a full on outage in the past few years, but during major storms I have seen some flickering and other issues. I just looked at the readout on the UPS, and so far it's kicked in 5 times this year. I have a pretty sizable NAS (40TB) and a beefy workstation - the $200 I spent on an APC BR1500MS is well worth the protection.
Safety is another consideration. My parents live on their hobby farm a few hours away - they are the only house at the end of a mile long road with older power lines. I put a similar UPS to keep their alarm/phone/network online. If there's a medical alarm, fire, carbon monoxide, water leak, etc the alarm can still alert the monitoring company, their internet/VoIP landline can fail over to cellular, and we can get alerted that something is up. Again, the peace of mind alone is well worth the $200 (plus $100 for a battery every few years).
Forgot to add, I was living in Virginia during the June 2012 derecho blew through during a period of 100 degree weather (https://en.wikipedia.org/wiki/June_2012_North_American_derec...). I was without power for an entire week - even in a nice subdivision with buried power lines, in a city of 100K+ people. UPS won't help for that, but it did automatically shut down the NAS and kept my modem/wireless router going for a few hours. Bought a 2,000 watt inverter generator for a few hundred dollars. If you have somewhere to store it, highly recommend purchasing one. Extremely quiet (~50db), sips gas (1 gallon/12 hours), and enough power to keep a fridge, small window AC, and lights going. Even managed to run the dishwasher. https://wenproducts.com/collections/inverter-generators/prod....
Canada here, metropolitan area. Power outages happen about once a decade, usually as a result of a wind storm. I have had zero server downtime due to power outages, but UPS / battery failures occur at about 4x the outage rate.
Agreed, compared to most developed countries, a couple times/year would be extremely high. But, it's not relatively high for America. I remember living in the midwest and every major storm usually resulted in power lost to somebody. Many of my neighbors had (Generac) generators that would kick in automatically when the power went out. They're so common there that home depot stocks them on the floor.
In Southern California, we rarely get rain, let alone major storms and we still lose power 1-2 times/year.
On the other hand, I lived in a number of areas across Canada and power outages were quite rare despite the harsher climate.
My power goes out all the time in Denver. In the summer, with no adverse weather. The grid here is a joke compared to most of Florida which is FAR more resilient.
I have power failures a few times a week up to a few hours each time, living in a capital city of an European Union country. Having power failures a couple of times per year is "extremely infrequent".
Here in San Francisco's Mission district the power used to go out every time it rained heavily (2-3 times a year for a period of 5 years or so.) Seems to have been fixed a year or so back. Heavy rain in SF isn't even all that heavy, though the roads do become almost impassable since they're not built with a gradient.
In other cities I've lived in power cuts were a once every two or three years thing and you could still drive on the roads when it rained.
When our office was in Santa Monica, we'd get short (10 second - 2 minute) power outages several times per year (5-10 times), and real 1/2 day long outages about once or twice per year. Several days we were told to go work from home instead.
The article correctly and importantly points out that there are many types of UPS. Most recently, I got a production-grade APC SMX1500RM2U for home.
Another thing to be aware of, if your PC has a PFC power supply, is that it might reboot or have undefined behavior on event of the switchover to battery, unless the UPS is designed to avoid this.
The cheap UPSs aren't very good and don't work well or at all with modern PCs. I've had several of these from Cyberpower that haven't let me down. Far more reasonably priced than APCs equivalents.
Same here, I've had no issues with the 1500VA version. It's line interactive so should still work if the battery dies, but I can't say that I've tested it.
I don’t know if their newer revisions are any better, but I bought a CP1500PFCLCD in 2011. When the battery died from old age, it cut power to all of the battery backed receptacles until I power cycled the UPS. When the charger failed a few years after that, it did the exact same thing.
Seconding Cyberpower. Even their cheaper ones do better than BackUPS and don't fail to pass mains power along if the battery fails a check. The PFC Sinewave you linked do some nice power conditioning. It's a big help for me when my house switches to backup generator.
(Why yes, I live in California, land of multi-day power outages. Why do you ask?)
I like Cyberpower for home A/V racks, and yes, that PFC Sinewave line.
Generally the rack mount models, such as CyberPower OR1500PFCRT2U PFC Sinewave UPS System, 1500VA/1050W, 8 Outlets, AVR, 2U or CyberPower OR2200PFCRT2U PFC Sinewave UPS System, 2000VA/1540W, 8 Outlets, AVR, 2U.
Very happy, and stopped having to replace sensitive flat screens in bad power zip codes (trees vs. lines every other storm, such as for the last week after Isaias).
FYI for the APC Back-UPS "Pro" - the "Pro" model's topology is line interactive. (Or at least my Australian 1500 version is, as well as the other "Pro" models I checked.)
I recall from my years as a small business sysadmin that some of the Smart-UPS systems that have a serial connector for management access require a special serial cable. If you plug in a “standard” serial cable it cuts power to the whole UPS unit and your whole server rack loses power even if the batteries are fully charged. That was a bad day.
Indeed. I, too, learned that lesson the hard way many, many years ago! I also know several other folks who learned that the same way that you and I did.
Although I'm a bit "old school" and generally prefer serial (RS-232) ports, this is one case where the change to USB really was a huge improvement!
I did a lot of research before I bought a UPS. The constant that I saw across various brands was that these batteries will die after a few years without any warning. They have to be replaced proactively. No if’s and’s or but’s.
My experience has been that inexpensive UPS units create more problems than they solve in the long-run. Spending more money up front to get a corporate-production-grade unit is worth the money.
This reminded me of James Hamilton's (the AWS datacentre guru) writeup of the power outage at the 2013 SuperBowl. Specifically how the default configuration of backup generators is often to protect themselves, even if that means shutting down the things they are meant to keep running. In my own experience running a reasonably sized datacentre I was often surprised at how UPSs and generators reacted to adverse conditions. One thing with power that you don't want is to be surprised. But it's difficult to test this equipment so it's mostly a learn as you go experience.
Electrical Engineers would determine the settings for the electrical protection of a generator. Mechanical Engineers would determine the settings for mechanical protection of a motor. To operate the equipment beyond thresholds would result in reduced lifespan or immediate and permanent damage to the equipment. If this is what the customer requires they have to specify that to the Engineer so that when the equipment blows up the Engineer is not liable.
Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.
Diesel generators for hospitals and water pumps for fire suppression systems
Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
True, but the easiest test is to pull the main power to the hospital and watch the system automatically take over. If anything fails you know what is wrong. This is scheduled so that those in surgery (or other worrisome situations) have extra help and can signal get back to mains power now.
ideally the hospital’s generator control system or transfer switch is able to start and synchronize the generators and turn up the throttle until no power is being imported from the utility prior to opening the utility feed, thereby transferring the load from the utility to the generators. This test is only performed when there are no surgeries scheduled and it is cancelled if there are any emergencies coming in all of the sudden. If the test is already in progress incoming ambulances with patients requiring surgery are diverted to another hospital.
To finish the test the hospital is synchronized to the utility, the throttle on the generators turned down until no power is flowing through the generator breaker which is then opened.
I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
That is a good way to test the generator without interruption, but at some point you still need to test the ATS under real world circumstances: unstable or suddenly missing utility power. Throwing the main covers one of those, and the other requires a little more planning and resources.
Honestly I am struggling to remember an instance where we purposefully plunged an operating hospital in to darkness.
You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators.
If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should.
Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.
You can (and should) do all that. However until you actually disconnect from the utility power you don't know for sure that the systems all work. The generator powers a load, but turns out your hospital is more load than you thought. Or the generator starts but the switch to connect the hospital to the generator is broke and you didn't test that...
In the end you WILL test what happens when the hospital plunges into darkness. Would you prefer the first test to be a a bright sunny day when all the staff is ready for something (and you can turn the utility power back on quickly if something fails), or when someone with a backhoe/chainsaw has an accident with your power feed. The latter will happen, I don't know how or when, but at some point in your future trees will take down power lines, backhoes will cut something, major weather events will take out power for days, maybe a large blackout...
You should have course test everything separately often. However if you wouldn't let a random electrician (electrician only because they will know how to not kill themselves) turn off your utility connection with only a day of warning, your system isn't trusted to handle real events which are typically much worse than a clean turn off the switch.
I remember some complication with the elevators not working or homing properly at one place so we must have had an outage. it was probably at 3am, scheduled, and with incoming patients diverted. All our tests are and I’m in the electrical room so they all looks the same to me. An actual outage to vital loads would be done once and then you shut the door and walk away as there was never any question that it was going to work.
FYI where I’ve worked the tiers of loads are metered individually and picked up in order of priority and with a check to verify the combined capacity of the generators which are online is great enough to pick up the load. Likewise loads are shed in order if there isn’t enough generation such as one generator tripping.
I don’t really see that much difference between opening a breaker and bad quality power. It is the same sensing and logic in both cases: is the magnitude and frequency of the phase to phase voltages in the acceptable range or not?
The important part is that whatever bad happens to mains power, the backup systems detect it and properly take over. Bad frequency or power out are both bad situations that the backups need to handle. You need to have confidence that your systems will handle both. If you only pull power and never test bad frequency, one phase missing, brown outs, over voltage, (list not complete because I don't know everything) then you have not done your job. If engineering determines that your test also covers another bad scenario then I'm fine with not testing it.
Electric utilities provide appropriately higher levels of service and communication to hospitals. Hence there are protocols insuring hospitals don’t lose power in the way you are imagining. The people involved are serious competent professionals with extensive relevant experience, training, and education focused on safety and in touch with the specific facts of actual hospitals and utility grids.
There isn't much that they can do about storms knocking down wires, squirrels chewing a wire and knocking out a phase (other than taking down all phases). Which is why hospitals have several backup systems.
Some military equipment, especially naval, has something called a "battleshort" where electrical protections can be disabled during combat, where losing equipment could be more dangerous than electrical hazards. It's interesting to think about whether this could be applied to DC environments, but honestly, I think few DCs actually have this kind of risk aversion for downtime. Alternatives like precautionary replacement of generators that have been oversped probably come at too high of a cost, not to mention that many of those false-positive-prone protective devices are there to reduce risk of fire.
Indeed. I forget where I read it, but it provides that power increase for (IIRC) some small double digits number of seconds before you need to throttle back, or the engine eats itself. If you are able to make it back to base, and that WEP wire is broken, the engine needs a rebuild before it flies again. Definitely an impressive power boost though.
The MiG-25 also have something similar where the engine redline is marked around Mach 2.8, but there's no system stopping the pilot from keeping it maxed to somewhere above Mach 3 but damaging the engine in the process.
Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster
No, SR-71s are still limited by engine temperature. They measure the Compressor Inlet Temperature, but I suspect that actual limitation is the turbine inlet temp. That said, it'll merrily try to go above that without issue if you don't throttle back.
Additionally, there's the Mach Cone. The shock wave off the nose forms a cone shape, with the angle determined by the speed of the aircraft. From the nose to the wingtip forms an angle of about 17.5 degrees, which corresponds well with the max speed from the CIT of Mach 3.3.
That both methods of determining max speed agree shouldn't be surprising. Skunk Works was filled with good engineers.
MiG-31 has limiter on engine controls connected to airspeed and mach number indicators, because the engine will happily run above Ma 3, with no issues.
The airframe, however, will get bent from temperature and you better hope the missile cooling loops don't fail.
The power isn't dialed down at lower speeds in order to provide thrust for maneuvering and climb.
We use something similar on turbocharged drag (grudge) racing cars.
The boost controller has a "scramble" button which overrides the preset boost targets and allows you to run a higher (and unsafe) boost level as long as the button is pressed.
This is useful when you're in the middle of a $10,000 race and you're losing by a few feet.
All motorcycle mechanics have a sign "speed costs money, how fast do you want to go". It isn't a joke, most of them can get as much power out of a 500cc motor as industrial engines get out of 19 liters. The catch is that you have to rebuild the engine every 10 hours of operation, while the industrial engine will run non-stop for many years.
Sure they do, run away engines are a real problem. They will sometimes start using their own oil as fuel, and at that point you better run. (modern engines have safety air shutdowns)
Yes, definitely real! Although what you're probably thinking of is nitrous oxide injection. One popular manufacturer of nitrous oxide systems is "NOS", which was featured heavily on the Fast & Furious movies.
The "scramble" button I'm talking about is for "boost", which is the amount of positive air pressure inside the intake manifold.
Both "boost" and nitrous can dramatically increase power by essentially getting more oxygen into the combustion chamber, but nitrous is a much more dramatic effect.
A320 and B737 had special thrust bump modes for short fields like KSNA. Going back many years to the P&W JT8D you could operate in the red provided you did it for only a certain amount of seconds and logged it down.
There's a maximum power mode selectable on the FADEC interface, which gives you at least 30 minutes of engine life - any use means the engine needs total overhaul.
This mode is pretty much reserved for go around or post V1 takeoff when one engine on twin engined airliner had failed.
Many years back I had the opportunity to visit one of those data centres at the heart of Europe's financial operations. In case of fire, you had a minute or so to get out, because then the place is pumped full of argon. I guess that's a DC variation of your "battleshort"...
What would they do in case of injured/incapacitated people in there? Seems conceivable that some things that could start a fire might cause that as well.
Interesting, one of the guys that worked on my Uni's DC told us that the fire suppressor system is interlocked: to enter the DC, you need to open a door using a key. Upon opening the door, the fire extinguishing system is disabled, and is only enabled when the door is closed. The people working in the DC are supposed to keep the key on themselves. Also, they used Halon 1301 instead of Argon as the gas.
Amusingly, tetrafluoroethane and pentafluoroethane are themselves being phased out in refrigeration applications because of their sky-high global warming potential, thousands of times worse than CO2.
Yeah, you can't install new Halon 1211 systems, but it's still out there. New systems are usually FM-200 or Inergen, which I believe can use the same plumbing as Halon.
Older systems, before Halon was invented, are typically CO2 or Argon. They require a lot more agent to accomplish the job, but they're very simple.
I worked briefly in a CO2 datacenter. It dated from the mainframe era, and had some gorgeous mementos on the walls. There were SCBA packs in the hallway outside, and everyone got a brief safety lecture before being allowed into the facility. The point I remember was basically "The alarm will sound intermittently for twenty seconds, then go solid for five, before discharge. If it goes off, do not stop to save your files, just run for the door. If you're not outside yet when the solid tone sounds, hold your breath and run on what's in your lungs."
Apparently the CO2 is considerably nastier than Halon to get a lung-full of...
Yes, carbon dioxide is not biologically inert like Halon is. It's the signal your body uses to say it's out of oxygen. You will definitely react negatively to getting a lung full of it.
For reference: It's not a lack of oxygen that causes the sensation of asphyxiation. It's a surplus of CO2. It'd essentially be the gas-based equivalent of waterboarding.
Worked on a project in late 80s for an Italian bank. We implemented a SWIFT (financial) interface (on Stratus for reliability reasons). A few fun and games:
- The Stratus supposedly had fault tolerant duplexing of boards - pull one out and the other keeps working. A famous demo where local consultant pulls out the only board which wasn't duplicated!
- The machine room was also fire proofed with Halon gas (if I recall correctly), but due to security concerns the door was locked from inside by operators when present (luckily as programmers we weren't expected to work inside with doors locked!). Luckily also, it was my first exposure to "paired programming"/"paired operating" as in there were invariably 2 of them.
- A couple of the operators became too interested in how to work things, and wrote a quite comprehensive manual. They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves".
- the whole headquarters building was in a guarded compound, where ultimate protection against "red brigade terrorists" (https://en.wikipedia.org/wiki/Red_Brigades) were the armed guards. We used to meet them in the ground floor cafe every mid-morning, with a revolver on their hip, while they drank an espresso with brandy chaser. Really inspired confidence that they would be there to protect us should the need arise!
> They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves"
And this is why we cannot have nice IT security practices.
Most of military equipment is built to be able to operate outside the safety bounds, because the enemy in a battle is actively trying to push it outside these safety bounds and outright destroy it. Burning down your afterburner chamber and winning a fight is better that preserving it and losing the whole aircraft.
In a datacenter though your bets are usually much lower. Nobody is going to die, or lose many millions, if some of your equipment shuts down to prevent it from being damaged, catching a fire, etc.
There are though high-stake situations where you want exactly that: spend the entire amount of the resources of certain hardware to prevent a loss of life, or of untold millions, when you are powering a surgery chamber, or a large stock trading operation.
People who realize that do over-provision and pay top dollar for that when they can afford that. I remember that when a major fire occurred in one of the skyscrapers in the financial district of NYC, traders of a particular financial company were evacuated with their laptops into helicopters on the roof, and ferried to a spare office across Hudson river, in NJ. To minimize the impact of that, they connected to the corporate network via their phones as hotspots, and kept trading while airborne.
Of course one cannot hope to pull such an operation off without extensive preparation and likely regular drills.
Not preparing to a black swan event during a high-stakes event, like translation of a Superbowl match, sounds like either not having enough paranoia which is professionally required, or, more likely, as a cost-cutting after a wrong assessment of risks.
That sounds a lot harder than having half of each team in two separate buildings. If one needs to shut down the secondary team just takes the full load...
> (2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.
Day traders are expensive? I thought a lot of those folks keep what they kill, and when they screw up they don't get to eat. "Coffee for closers," etc.
> According to Allied Ordnance Publication AOP-38-3,[1] a NATO publication, a battleshort is "The capability to bypass certain safety features in a system to ensure completion of the mission without interruption due to the safety feature." It also says, "Examples of bypassed safety features are circuit overload protection, and protection against overheating".
> For example, the electrical drives to elevate and traverse the guns of a combat warship may have "battleshort" fuses, which are simply copper bars of the correct size to fit the fuse holders, as failure to return fire in a combat situation is a greater threat to the ship and crew than damaging or overheating the electrical motors.
> Battleshorts have been used in some non-combat situations as well, including the Firing Room/Mission Control spaces at NASA during the manned Apollo missions — specifically the Moon landings.
As a side note, with great apology to those who may have lost their lives due to the consequences of real-world battleshorts - this is actually a really interesting analogy for the "war room" scenarios we find ourselves in at startups. Risking damage to morale and productivity can be acceptable if the alternative is irrecoverable loss of the startup's reputation. But this also can't be a sustained state of affairs - these types of procedures are meant for a battle, and risk compounds if you don't return to a steady state. Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
> Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
... And 6 months later marketing people will have commandeered it, without regard to the original meaning, and they’ll be asking people to battleshort so they can really be agile and lean in on providing air cover via the updated sales deck.
Doubly so because I used to be in the US military but now live in a different country, and it's crazy to see how those terms and acronyms infect a lot of US business culture.
A decade ago I would have been fine, even glad, that I could wrap myself in verbiage that would jive with poor career choices I made at 18. Now it just feels incongruous or jingoistic.
Electrical codes in the US are very clear that they care much, much more about preventing fires than preventing loss of power. The one exception is the fire pump that provides water pressure to the sprinkler system, since if that's running there's already a fire.
The owner of the datacenter's aversion to downtime is not going to be a concern to the building inspector or the fire marshal.
Fire supression systems basically run their own electrical code.
At one of my previous workplaces I learned a bit about our local firecode in relation to electrical code. Mostly that firecode supersedes electrical code. The fire alarm is running of a 230V line (standard) without grounding (non-standard) via a special line from the upstream power transformer that has higher amperage tolerance.
All the fire detectors are powered and wired from this at 40-70V, no ground or if there is a ground, it's grounded at the control panel of the fire alarm system.
In the event of a fire, the entire system is rated about 10X over it's standard ratings. The wires that are allowed to handle 1 Amp before are now allowed to go to 10 Amps, just in case there is a short this might burn it out and allow other equipment to operate.
I've seen some systems that are "optionally fused", they have an internal relais that's held by external power and if the fused lines fail, it automatically switches to unfused power, where the alarm system can evaluate the condition and decide to go back to the fused line if power returns. That's on top of having it's own backup batteries. Some of the more modern systems have watchdogs systems built in so that in case the fire alarm computer is on fire, it won't switch to the unfused line once the computer is no longer operational.
I think you will find that those circuits use isolation transformers and have no path to earth normally -- if you touch active and normal building earth there is no circuit.
At to amperage ratings, don't you just mean that they have a 10x safety factor? Probably using current limiting supplies that can handle a short circuit. Fire circuits (and the feed from the main panel) should use fire rated cabling -- generally mineral insulated copper sheathed cables. Most of the standards (NFPA 75/76) are moving away from being prescriptive to being risk or performance based though, so it is possible they used normal conduit if the overall risk was small.
Incidentally, hospitals also make extensive use of isolation transformers for wet areas (like emergency / ICU), to provide a local point for resetting breakers (electronically), identifying which point is tripping, etc, see IEEE 602-2007. Much better filtering for medical equipment too. You also tend to see positive retention plugs (in the US, green dot UL 817).
The cables have fairly high ratings but in the even of a short circuit, the ratings have a safety margin of about 10x that the cable can last about 6 hours under, above the normal rating under standard conditions (ie, not fire).
I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems.
The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline).
Test components individually - at the factory and in the field - and test systems end-to-end.
Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope.
When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system?
Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails".
If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?
Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)
How do you test diesel generators which back up a nuclear reactor? I understand you do it during the plant downtime, but don’t you require something at the other end of the network that consumes the 1GW you generate?
The diesel generators can back up power to safety systems, so closer to MWs than GWs. Some run with left-handed and right-handed engines coupled face-to-face for redundancy; thus you might AFAIK shut down fuel injection to one to prove the other is supplying torque.
A combination of routing the power elsewhere in the grid when possible and banks of gigantic (shipping container sized) dummy loads. As other people have noted, you don't need to dissipate the full amount of power all at once, you test the generators individually.
You test them live: someone turns off the outside power switch and you watch them start and take over automatically. Everyone is watching the gauges and if there is a sign of trouble you reengage mains power, then fix the problem and retry. This test is done monthly.
I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.
The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.
Most of the COs I worked in did monthly generator "dry runs" without putting load on it, and then every 6 months they'd do a "drawdown test":
Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas.
After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power.
Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back.
When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank.
Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
That's a good point about testing. I think the key point is where the testing is happening. I'm sure the UPS (and other power equipment) is tested by the manufacturer for many different failure scenarios. The issue is that to the end user, the response is often surprising. So I'm sure to the APC engineer, having the unit shut down when the battery fails (according to some metric) but while there is still mains power available is something that they tested. But the consumer hasn't tested a battery failure, and in fact it may be difficult to test the exact failure mode (low voltage etc) without access to some spare, old, batteries. So when it happens and you're surprised and you call the manufacturer, they (almost always) will say that it operated according to how it was designed and tested. It just didn't do what you wanted or expected. At least that was always my experience doing a post mortem on a datacentre outage.
My grade school neighbored a hospital. I recall occasional loud rumblings of the backup generators during recess, presumably they were tests. Legend was they came from a WWII submarine! Neat!
Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online.
I never did get a chance to see the generator room.
e: Another personal anecdote that is more relevant to the topic.
I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.
Working in a 911 call center this is how it worked for us. Except we had larger UPS's that could last probably 30-60 minutes in the building and then a large external generator in a small building beside. You'd see stuff like the lights that weren't on the UPS's turn off for a few seconds before the main generator fired up.
Everything involved in 911 calls had special tagging to indicate it. Failure to go through appropriate review and approval processes before doing work on tagged equipment would create a Resume Generation Event.
My current job refuses to replace old UPSs that have died, because they have a generator to keep the building powered in case of loss of power. But that doesn't keep your PC running for the moments it takes the generator to spin up.
Oh yeah, if we had that where we worked people would definitely be in danger. Most likely all of our systems would be down if we didn't have the UPS+Generator setup. It often takes 5-30 seconds for the generator to kick in and be running at full capacity. Normally 5 seconds, but sometimes there is a bit of a delay.
The other part of the equation is that they're recommending that everybody switch to laptops, which are capable of running for a long time without power. My job involves a lot of long compiles, so I want to stick with a desktop PC to get the maximum compile speed.
Considering how old the software we used was it definitely could run well on a laptop. However it does mean you have to manage the battery life on a lot of different laptops versus one central area.
Don't get me wrong, the company pushes laptops for reasons other than eliminating UPSs. Even a nearly dead laptop battery should tide you over for the minute it takes for the building generator to get online. I don't think managing laptop batteries is a big concern for our IT department.
That blog is an amazing insight into internet operations + culture from that time frame. Windows Media for streaming! URLS with perl/script.pl in them! AIM/ICQ! It helps the writer has a bit of a flair for storytelling.
WWII submarine generators produced DC, so they couldn't have been used directly as backup power for a hospital. Maybe they were refitted with alternators.
Nowadays, some hospitals, especially those with heating loads, are doing the opposite:
Run a combined-cycle natural gas generator 24/7, and use the grid as your backup. Gets good efficiency. Runs loops on the coolant, oil and exhaust for water/building heat, and can switch over instantly in the event of failure.
Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?
Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand.
But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.)
If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage.
But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
You can put your equipment on a voltage relay to make failure modes as simple as flipping a circuit breaker. It will turn off once voltage drops too much and will wait for it to stabilize for specified amount of time to turn it on.
It depends. We built and deployed some IoT devices (controlling and reporting status of lights, HVAC systems, that sort of thing) to a bunch of greenhouses--one of which ran on generator power.
The customer was running the generators right at the maximum capacity, and we'd see really, really odd things on the power lines. Some times a surge like you mention, sometimes changes to the frequency, sometimes changes to the voltage. We discovered all sorts of weird brownout conditions in our devices.
It's difficult in practice to transfer load to generators for test especially if there's load shedding involved as is common in larger facilities (higher-power-consumption items like part of air conditioning capacity are cut off when transferred to generators). To help address this a lot of more critical facilities are equipped with a test load (really just a very large outdoor resistor with active cooling) that allows for running the generators at rated load for test and exercise without actually transferring. Of course, these tests do not always detect problems with the transfer equipment, and if things aren't designed and maintained well the generator may experience load under real conditions that behaves differently from the test load.
Iirc (top of my head), some datacenters go off the grid regularly at peak (provider) load times and get more favorable rates for doing it. I guess it ensures your fuel doesn't go stale and all your transfer systems work.
Good point. Many power companies will give you a nice discount year round if you will disconnect from mains on demand. If you are on the fence about a good backup generator this will more than make up the difference.
In our small data centre we have a board with five lights and a key. You turn the key and the mains is cut off. The lights give the status. The UPSs beep for a few seconds "on battery", the genny fires up in the boiler room and when it stabilises, it takes the load. We leave it running for 5-15 minutes, go out and check the fuel levels and top up accordingly. It's diesel so safe to hot refuel. When done, turn the key back and then the UPSs take the load again and then hand back to the mains. The genny runs for another 15 minutes and then shuts down. That should avoid flip flopping.
You then fill in the test form on our wiki and job done. We do it weekly to fortnightly. An ESP8266 based thingie reports status back to the central monitoring (GPIOs and ESPHome -> Home Assistant)
After reading this, I checked what topology my UPS is using and was happy to see it's line interactive. I have two CyberPower CP1500PFCLCD units, and am approaching the one year mark. It looks like the batteries are even hot-swappable, which I'm sure will come in handy when replacing them.
I also have an APC Back-UPS. Mine are the Back-UPS 650 model, I have two of them. Both emit an annoying LOUD high-frequency capacitor whirring noise that is audible to my ears even when the loud AC intake fan is blowing in the same room. I'm actually shocked any company would ship a product in this condition, unless all their customers have hearing problems. It's a power strip... it's supposed to be silent. Not like a computer.
If my batteries die, I'll just replace them with non-battery backup strips. Not worth the trouble for me living in a city where power goes out maybe once per year on average for 10 minutes, and I don't have servers that need to stay online anymore.
But the main point in the article is worthy: UPS manufacturers have been skimping somewhat on functionality, features and quality over the past few years...
No, the article is wrong: the low end UPS devices are bad on quality and functionality and lead-acid batteries in UPS devices is a lot shorter than some people would expect.
I still have a few APC Smart UPS's bought in 1999-2000 running in my house; at the same time I had a few cheapies that broke after a few years, sometimes in spectacular ways (electronic exploded melting part of the battery case); I will not name brands, it is not worth.
Since 2000 I changed cases full of batteries, usually every 2-3 years. The good thing with the APC Smart series is that it works even if it gives an alert that the battery is bad, but you will get a few minutes of power instead of 30-60 minutes (my server & NAS is very low power, 5% of the UPS capacity). I also set the server to shutdown when the battery is reaching 50%, so I never deplete the batteries. At the same time, the UPS on the water pump is usually running on batteries as much as it can, so I need to change these batteries after 2 years. I have both the server and the water pump in the basement, the temperature is around 16-20 degrees Celsius (winter-summer), when I was keeping the server and the UPS in the house the battery life was at half (~18 months). The heat and the deep discharge cycles are effectively killing batteries.
I had the same type of battery on my motorcycles, one lived about 8 years and a second one 6 years, but in very different conditions: it was warmer, but almost no discharges.
Deep discharge is a factor, but not a big one, heat has very little effect, it only affects voltages at which battery will be considered overcharged or deeply discharged as specs are usually all for room temperature.
I've used UPSes that discharge batteries no lower than 10.5 (no deep discharge) and only once in a couple of years and that charge them to and keep them at 13.6 volts under stable room temperature all year. All batteries lost most of the capacity after 3 years, two almost all of it, one had like 40% left. In five years only that one was still surviving short minutes outages, while initially was able to do a few hours. For comparison, I had one battery just laying around for 6 years in the same room without touching it and it lost only like 30%.
It turns out that APC intentionally overcharges the batteries to maximize runtime during the 2-year target battery lifetime, at the cost of killing the batteries relatively young. And the overcharged voltage drifts upwards over time.
Buy an automatic transfer switch. Plug one input into a UPS, plug the other into either the mains or another UPS (preferably a different brand but identical power). I resorted to this after having multiple APC and Cyberpower batteries fail killing power (even with a line-interactive APC). I run my network (cable modem, router, primary eero AP) off this, no servers. Means we can typically stay online even when the power glitches for an hour or two after a storm.
The one I bought uses standard NEMA cords and outlets: "CyberPower PDU15M10AT Metered ATS PDU" https://www.amazon.com/gp/product/B00NEHUX08/ — it’s overkill for my use case but was the smallest option I could find. I don't have any of the control modules (provide snmp/http APIs).
One of the annoying things I’ve seen with the Back-UPS range from APC is that they all seem to have an audible alarm when it’s running on battery that cannot be turned off. The UPS will beep periodically until you either turn the UPS off or until mains power returns. This is so annoying that you can find instructions and videos online on how to perform a surgery on the UPS to disconnect the buzzer from the rest of the circuitry.
Maybe APC really wants to drive users of its lower tier (and cheaper) UPSes mad with the beeps and force them to buy the more expensive Smart-UPS range that comes with a built-in option to turn off audible alarms.
Well, audible alarms do have a place in such systems to inform the person that the connected devices or systems are running on a limited battery capacity. But forcing that at all times is a big annoyance.
During an extended power outage years ago I used my Back-UPS to keep my phone charged while I watched Netflix over cellular. I ended up wrapping the thing in a whole bunch of blankets and towels to muffle the obnoxious beeping. Huge oversight that there isn't any button to silence the alarm, there are plenty of reasons someone would want to do it.
I meant to tear it apart and remove the beeper after that but never got around to it before the battery made it useless anyway. TBF it was a few years old at that point so the battery giving up wasn't unexpected.
"But forcing that at all times is a big annoyance."
Agreed. The cyberpower devices do let you "silence" the alarm, but even with the alarm "silenced" the UPS will begin rapid beeping as the battery nears depletion.
418 comments
[ 3.4 ms ] story [ 321 ms ] threadI've come to the conclusion that if you are going to use a UPS, your equipment should always be plugged in to at least 2 UPSs, either via redundant power supplies or an ATS PDU.
> So I thought I had covered all bases, and, indeed, the UPS proved useful on several occasions. I would quite often be on a work trip and receive an e-mail from the server informing me that mains power to the UPS had been lost, then another e-mail soon after informing me that mains power to the UPS had returned. Only once did the power cut last longer than the battery capacity, and the server was shutdown automatically.
It sounds like the UPS is doing exactly what he wants it to do. This is exactly why I bought a UPS. I would have power transients sometimes and rebooting my entire network (with NAS and servers) was annoying, and I didn't want it to go out when I was gone.
His main complaint is he bought a standby UPS where the battery failed, it turned off the UPS (which is understandably frustrating), and now replaces the battery every three years. Looking at the white paper referenced, it even says to go for Line Interactive for servers. I'm also curious how the person knew that power wasn't cut when he was away?
So other than buying the wrong type of UPS (which at face value looks to be a valid complain based on research)....why isn't a UPS as useful as they'd think?
I assume this would be because the power is always flowing through the battery/inverter? Wouldn't that kill the battery a lot faster? Although it'd make the switch from mains to actual battery power basically a non-event from the output's perspective.
Now that I think about it, going 120/240->low voltage dc->120/240->12/5/3.3 is incredibly silly. IIRC there are 48 (or -48?) DC power supplies out there, mostly for data center use. Does anyone make an ATX version that you could hook up to a UPS that has a DC output?
This is a design choice from APC, money is on the control electronics being powered off the battery meaning a dead or missing battery means the entire unit is non-functional. Making the unit resilient to this probably costs as much as a line-interactive unit, however, so you might as well just tell people to get a better unit than make a middle-ground solution.
This isn't common for servers though which, if equipped for redundant power, have two AC power supplies. I think this just comes down to the higher current draw usually associated with servers, running even 48VDC power can become costly and impractical when you need to supply say a couple thousand watts per rack, so the extra space and thermal load of an AC power supply in each unit becomes preferable.
The downside is that while redundant power supply is usually a "standard" feature on switches and routers (that is, they always have the connection whether or not you buy an RPS is up to you), you need to specify and pay extra for dual AC power supplies up-front in servers, so you don't get as smooth as an upgrade path. That said in datacenter environments you often buy servers, racks, and power systems all at once so it's not as much of a concern.
There's a similar tradeoff that exists around bottom-of-rack UPS and central (building or area) UPS---bottom-of-rack UPS tends to be more expensive, higher maintenance, higher thermal load, etc for large installations, so usually large installations use one (or two for A+B power) large central units which may be hybrid between different technologies like flywheel and battery (and you could view the transfer to generators as a "third stage" of a central UPS system), but it also makes your cabling a little more complicated and you have a bit of an "eggs in one basket" situation. Central UPS generally have internal redundancy so their risk of failure is low, and often a bypass device where if a non-recoverable failure of the UPS is detected a contactor "shunts" the entire UPS removing it from the circuit so you don't lose power due to a UPS failure, but that doesn't necessarily save you when the HVAC pours water directly into the UPS control cabinet causing widespread failure of the control electronics... a situation that I have somehow seen twice. I bet on bigger installs you can get an external bypass with some sort of health monitoring though? If HP/Tandem taught is one thing it's that you can always through more redundancy into your very special basket.
That UPS was well worth it to put a stop to that happening.
As a small business network/systems admin, I've had more failed UPS's bring things down than I can count. It's very common.
After wasting a lot of time remotely trying to diagnose this, it was obvious what was happening once I got on site. It turned out that once this server was shut down, the UPS didn't have enough power draw to power the other equipment attached to it, so a switch that was plugged into the same UPS would shut down as well. For some reason it had failed in such a way that it required a certain level of draw for it to function at all.
Most homes have something with a mains-powered, non-battery-backed clock. You get home and the microwave or the oven is flashing 12:00 and you know there was a power cut.
> So other than buying the wrong type of UPS
This isn't a failure of 'type' of the UPS - it's a design fault deliberately left in by the device manufacturer for market segmentation.
It would be trivial for them to keep the power outputs on if the battery failed but the mains remained on. Leaving it out is a business decision, hobbling their home product so it doesn't steal market share from their small-business product.
You really don’t want line-interactive gear when dealing with larger loads either, as the transformer will just cause extra current draw during a voltage sag. This is why you don’t see (many of) them above 3-5KVA.
These low-end APC dinosaurs still use lead acid, and in my experience (with a solid utility over the past decade) outages are just as likely to be caused by battery failure than some kind of actual power outage.
If anyone wants to build something better, happy to chat further.
Edit: Since we're armchair-building for a predictable, high-uptime application, failure modes might have an interesting advantage though. Any battery engineers know if lithium ion has more predictable or measurable failure modes? Yes, we've all seen the exploding hoverboards... I'm assuming theoretically well-built electronics.
Additionally, li-ion has non-neglegable disposal costs, so I don't think it's fair to count that solely again lead acid batteries.
Car batteries are lead acid/gel cell batteries. They last for years and can take a very large discharge while still being rechargable.
Also, by cycle I meant a single vharge-discharge cycle can be more useful when larger amperage or more power is needed between charges.
What I don't understand is why it would be so hard to just scale those power packs up a little bit and slap a normal 110 volt A/C plug on them? Wouldn't that essentially be a super efficient, light weight UPS?
I did a bit more digging and I found a couple of portable power packs that offer this sort of functionality: https://www.nytimes.com/wirecutter/reviews/the-best-portable...
What I don't understand is why they are so expensive, and why they can't seem to scale them up to the same size as a normal lead acid battery UPS?
It's good for brief outages caused by thunderstorms, though, so still better than nothing.
I have been using the same offline UPS and battery for the last 7 years and it is still working fine - a few days ago it handled a 15 min power outage with 50% of the battery.
Can someone explain why offline UPS can fail if the battery dies? Is it by design or a manufacturer specific issue (e.g. only APC brand ones do this)?
How did he also know power didn't go out too? That confused me as well.
I had a script in a raspberry pooling my UPS each minute asking if the unit was running on AC or battery, and dropping that info into a file when it changed. I know the software/daemon can also do something like that.
I actually had the exact same thing happen to me a few months ago. A power flicker so brief that it didn't even reset any of my clocks, but it caused one of my cheapo standby UPSes to give out.
"online" is generally a double conversion UPS.
These are generally the best for clean power to the electronics. Since its always supplied by the battery. Its also often one of the hardest on the battery.
Line-interactive are probably most common. They are pretty solid, but since there is a delay for the cutover, surges and really dirty spikes can make it through to the equipment. So say a lightning strike or REALLY bad surge on an overloaded generator can get through, whereas on a Double conversion it may just trip the fuse or breaker. Also running a whole home or standby generator on these line interactive can make them trip constantly since the generators often dont
A) run at 60Hz (ie: mine runs closer to 63Hz)
B) run with a pure sine wave
You can get inverter generators to help with this. But thats a cost too. And otherwise the solution is to "de-tune" the UPS sensitivity.
I de-tune this on mine. Its been fine, running on gennys during the rainy months for hours and every know and then...days at a time.
I have had to swap out power supplies more often..But even then, usually thats after like...5+ years of runtime.
I have to de-tune their sensitivity but they work.
But it does stress the CPU power supply caps a little more and shorten their life. But the caps on the board are fine, and havent had an equipment failure outside of a PSU...And ive run literal weeks on genny power.
But that stress is more due to Hz being at the threshold and artificial stepped sine wave i think. Voltage fluctuations when load kicks up certainly doesnt help.
The constant humming (no fan, just electrical) so cant sleep in same room
The loud can't turn it off beeps when power is out.
And none of those come in the product description.
I personally like the white noise from electronics, so I can't help you there.
It's enough of a pain to do that rather than configure all 3 UPS devices in my house, I've only ever gotten around to doing it on one.
Can't say I've noticed the electrical hum, except during a blackout. But I don't have mine in a bedroom.
That depends on the UPS, it seems.
I have a pair of Back-UPS 1500s with the external battery packs. They have a button on the front-panel for disabling/enabling the audible alarm (just hold it down for 2s, IIRC).
Now imagine the sound at the APC factory, when the power goes out and thousands of UPSes on the charging racks start beeping in sync...! (this is not a hypothetical)
Most tasks are running scrapers or data wrangling so I don't have to keep my desktop running.
I've been doing this for maybe close to two years, and for now it works. I wouldn't recommend for VMs though.
Also, your router needs an UPS too but it will be fine with a cheap one.
Why not? I have a laptop with a 4th gen i5 that has VT-x. I've upgraded it to 16 GB RAM and an SSD. For now, it's a Spotify connect player but I'm contemplating running a couple VMs on it too, like pi-hole and whatnot.
Sure, that’s still reasonable, but it’s not young.
I’m also using similar class hardware for my VM server at home. Works well enough for home workloads, and for everything else there’s the cloud on demand.
I wouldn't run VMs on it, but I do run docker on it, and that works just fine.
Also, RDP. I still hope for something so smooth in Linux, using console meanwhile.
I did some tests on a RPI3 a year or 2 ago and showing some scrolling test in a UWP app was making the whole machine lag and be unresponsive.
I've also imaged a version of that SD card, so it would be easy to just swap out and probably get things running again without much hassle.
I'm sure there are more fancy solutions with overlayfs, making the SD card read-only except for upgrades. Though I couldn't find a nice resource on it at the time.
It has only been down when the power has been out, and so far no issues with the card. It is however an A1 or A2 class card (I forget which), so supposedly design with applications in mind.
It should also be noted that a lot of issues with the Pi is due to poor USB cables / power supplies. I had another Pi that kept crashing every few days, until I measured the USB "charger" cable I was using for it. Turns out it had a resistance of almost exactly 1 Ohm. So if the Pi drew say 1A, that would be 1V loss in the cable... After I swapped cables it has been rock solid. A key point here is that a lot of SD cards do not like losing power while being written to. Avoid that and a quality SD card should have a long life.
Any suggestions? Though I'm a software engineer, I've worked mostly in app dev rather than infrastructure setup. This project is a way for me to learn more about the latter. The rpi is the easiest and cheapest option, but I have wondered if it wouldn't make more sense to use AWS or something like that. I wouldn't mind paying a few bucks a month, but more than that and I'd rather wait until I've learned a bit and know what my specific needs are. I've also considered an actual computer, but figure it's a waste of power when my initial work will probably just be figuring out the server config, HTML pages, etc.
You could host everything on your home network but there are a couple of things you should probably know before you start: 1) Some ISP do not give you static IP, meaning its extra work
2) Security - Ideally your home network has some kind of segregation, you keep your trusted devices on one, IoT or not patchable on another for example, with firewall blocking connections from dirty network to clean one. Having externally facing website potentially having direct access to personal devices and services might be an issue from security point of view.
You can sign up on AWS/Azure or any other cloud provider for free and get free credits, which might be enough for you to learn. Feel free to ask further questions if you need to
I have a running rpi for more than half a year. No issues whatsoever. But again - I have a very stable electricity and I dont remember last time it was cut not due to planned maintenance.
1) I think the pi is not particularly power efficient.
I got this impression from this (old) article:
https://www.bitwizard.nl/wiki/Reducing_power_consumption_of_...
2) I think the pi would benefit from a more robust filesytem layout such as an overlay filesystem to allow continuous writes to /var to go to ramdisk.
Openwrt has a layout like this and it prevents flash from being burned out and prevents most problems if power is abruptly lost.
While there might be efficiency gains available, RPIs use a very small amount of power compared to a laptop. Idling a RPI4 uses 2.8 watts[0], which not only rates favorably compared to most laptops, but is far below what my charging cell phone uses. Maxed out my RPI can only hit 15W, since that's the maximum power that my official USB C adapter can output.
Now that being said, it does depend on what kind of load you're expecting to handle. If you're constantly maxing out a RPI, it is probably more energy efficient to purchase a larger server than to just keep adding RPIs. But if you're staying well below the theoretical max of a RPI, it'll consume far less than a used laptop.
> I think the pi would benefit from a more robust filesytem layout such as an overlay filesystem to allow continuous writes to /var to go to ramdisk.
This is one of the biggest drawbacks of the RPI; microSD cards cannot handle a ton of writes without frying. I've settled on using log2ram to ensure that logs are only periodically flushed from RAM to the SD card, to extend the lifespan of my cards.
Watts per low traffic website it's very efficient. CPU is generally not pegged at 100%.
For watts per CPU cycle not as much, I agree. Say it takes 10W at full power (I've not measured). That makes it 1/10th of a normal CPU. But it's like 30-40 times slower.
They really think their prosumer workstation is going to give them an advantage.
I guess thats one silver lining about Robinhood’s proliferation, many people know its good enough. Gullible traders still misread options settlement UI but seems there is a selective evolution at play to make that less common too.
[1] Quite often, I couldn’t ssh into the pi for all kinds of reasons. With a laptop, I don’t have to worry about that.
Have you any suggestions for increasing the amount of storage one could address? I can't find any Thunderbolt 2 or FireWire enclosures so I'm sadly limited to the two USB3.0 ports. Right know my plan is to resort to putting a high-capacity 2.5" hdd inside the MBP and adding a usb hdd in addition to the one I already have. But that doesn't strike me as ideal for some reason.
USB 3 is fast enough for a hard drive.
https://old.reddit.com/r/spicypillows/
Relatively recent smartphones also work great for a lot of stuff.
Then the next level of slipping down the chain is the headless server or timelapse photo taker. :)
Refurbished with a Windows license (if I ever want to use it) was less than a surplus laptop. Power usage seems acceptable but I'll compare with a Kill-A-Watt. I have one on the way and if it works out I'll grab a few more.
I do like the idea of a surplus laptop cluster, mostly because of the built in UPS factor but there's a lot of wasted hardware there, such as the monitor and keyboard. But I suppose that's kind of a feature too.
I have shower thoughts of some kind of custom bladecenter made of old laptops but that's a lot of work for probably zero benefit.
Can you post a link to those 7 year old micro towers, please?
The tiny tower I have on the way is one of these: https://www.lenovo.com/us/en/desktops/thinkcentre/m-series-t...
They’re available in a bunch of configurations and on multiple sites so you can look around for a good deal that fits your needs. I found one with an SSD for additional power savings.
In regards of power efficiency I'd still be drooling over one of the Xeon D-1600 configurations but haven't found one that's not in a rack form factor yet.
https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E...
https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E...
Also available with AMD Epyc Embedded (35W for 8 cores).
Broadwell NUCs are available on eBay and Newegg has a matching Akasa Plato X case for $100. With an SSD, that has no moving parts. But only dual core, single NIC.
In most cases the beefiest air cooler + the lowest tolerable fan curve (meaning no overheating) will do the job just fine, especially if the CPU TDP is between 35-65W. Alternatively you can also limit the CPU power usage by disabling turbo or doing other tricks that forces it to run slower, but more efficiently.
Some fanless cases here for different NUCs: https://www.neweggbusiness.com/product/productlist.aspx?subm...
Plex can use nVidia cards for hardware acceleration for encoding.
These features do require an active Plex Pass.
https://support.plex.tv/articles/115002178853-using-hardware...
Replace the fan with a Noctua-type quiet model bolted on. If the HSF needs that airflow directed, hack a simple shroud (or 3D print a replacement.)
I run a uSFF HP box for my hypervisor and a low power J5005 core in my NAS and am more than happy with the performance let alone minimal power usage. Sure the J5005 system takes ages to update requiring a backup DC/PiHole setup on the HyperVisor but it just works at low cost.
I thought I should replace my 2012 Mac Mini (HTPC) with a modern Hackintosh and 6c/12t or 8c/16t and do my testing workloads on that (or even just use my current 2c/4t for small workloads). But it's just so easy to spin up DO droplets within seconds and do so programmatically. Plus you can snapshot and create droplets from those, and turn on backups if you need to.
It's kind of annoying and I'd love to see a solution that isn't just "get a battery based UPS" -- maybe something capacitor based, enough for say 15-30 seconds?
I guess I could get a flywheel [0], but I don't think it'd fit in my basement.
[0] https://www.vertiv.com/en-us/products/brands/liebert/?id=59
The conditioner provided a constant voltage to the servers and could correct "sags" where the voltage would drop, and it could keep things going for a few seconds if the power went off completely. As you say this took our total outages down from probably 10 a year to maybe 1 or 2 - most outages are quite short. Although looking at the graphs on this thing, there were pretty frequent surges and sags coming from the street, like when you see light bulbs flicker in your house.
One other thing we noticed was that if there was a short interruption, all of the HP servers could survive for a second but the Dells turned off right away. Different power supplies store up some amount of power locally.
https://new.abb.com/ups/power-and-voltage-conditioners/volta...
https://www.atx-upsu.com/
Looks like it's capacitor-based, removes the need for an inverter, and even has a clever way of communicating to the host, by taking over the power switch connector.
Unfortunately, I can't seem to find a price or where to buy. Would love to see more momentum around products like this, and marketing directed at home systems.
EDIT: Thought of one more system that uses capacitors to limp to a clean shutdown--the Unifi Cloud Key Gen2.
Again though, the power solution isn't packaged for consumer use.
Though, maybe it'd be amusing to try to hack together a NAS out of Cloud Key Gen2 Pluses... there's a hard drive bay in each one ;D
For once there was a high voltage fault in my neighbourhood, destroying few lights, all mobile chargers plugged in and microwave left in standby. Had I not been home, and fault lasting more than 5min, things might have gone really bad.
Now on, Whenever I leave my home for long duration, I just empty the Refrigerator and switch the house mains supply off before loving the house.
First and foremost: fire alarms. Buy them, and use plenty of them.
Secondly: let your neighbors know you’re on vacation. This implies a certain amount of trust with your neighbors, but it’s good to be on good terms with them for more reasons than just this.
Thirdly: I get notifications if a server goes down or a fire alarm is triggered.
With all this in place, I don’t worry at all. I may be somewhat ignorant or oblivious to certain risks, but in general I trust the safety net I just described.
- Sustained overvoltage. Any overvoltage lasting too long to be handled by a surge protector should trip the main breaker.
- Sustained undervoltage. Some devices don’t like this.
- Phase error. A two phase main should detect incorrect phase-to-neutral voltages and incorrect phase-to-phase voltages. A three phase main should also detect incorrect phase sequence. This should also catch some cases of a disconnected neutral.
- Excessive neutral to ground current. In the US, for reasons I personally strongly dislike, every “service” has a connection from ground to neutral, usually in the meter box or the main panel. Various errors can cause substantial current to flow through this connection and can be dangerous. For example, if a house and the utility both have excellent grounds but the house’s neutral feed breaks, all the voltages can be close to correct and the house’s neutral current will return to the utility through ground. This is dangerous and will not be detected by common equipment. (A whole-house residual current detector would satisfy this, too. I think these are used in Europe.)
I used to feel this way, but then I realized that the electronic devices don't know I'm away on vacation, or when I've returned, and they're just as likely to start a fire when I'm down the street getting coffee as on the other side of the planet. And when I return, my presence doesn't magically reset some catastrophe timer and everything's all better because I touched "base."
That said, if I'm gone from home long enough that I turn off (or down) the refrigerator, then I'll unplug everything. So I guess I'm still a bit paranoid.
A lot of devices will smoke before starting a fire, sometimes for quite a while until temps increase sufficiently for ignition. You would smell the smoke and figure out the culprit.
Turning off everything is still worthwhile while you are away. This is just risk management. It is a small risk (maybe you have faulty appliances or house wiring? maybe you live in an earthquake or tornado area? etc) but the mitigation is also not very inconvenient.
That said, if you can have a smoke detector pinging you (or a similar device - Amazon's cameras have a microphone that can alert you to alarms) it would be better. As you point out, stuff could happen while you are temporarily outside the home.
Responding to a water-related disaster is a different story, though, and worth bringing up.
You might want to consider turning off your water at the main if you're going to be away from home for an extended period. I was fortunate in that the only water-related failure I've had personally was a ruptured water heater leaking into a floor drain of a 1st floor utility room. I know others who haven't been so lucky (having clothes washer hoses rupture, fixtures fail, etc) and have had water running inside their home for an extended period (several days, in one case). Until someone notices the local municipality reading excessive usage, a neighbor noticing water running out a patio door, etc) the damage continues and, to add insult to injury, you're also getting charged for the water / sewer usage.
If you believe fires start at random, there's no difference between a 2 week holiday and 9 weeks working 8 hours a day.
Here is how I differentiate the APC brand:
* Back-UPS: inverter offline, expect a 15-20ms power transfer
* Smart-UPS: interactive, inverter is always online 2-5ms trasnfter
* Symmetra and above: inverter online, voltage conditioned, 0ms transfer
Most desktops will work just fine with the Back-UPS line. The processor is practically sleeping. Workstations should be on a Smart-UPS because the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
I have these.
https://www.apc.com/shop/us/en/products/Power-Saving-Back-UP...
They are line-interactive. Most of their "tower" UPS models seem to be Line Interactive.
Its those little ones that look like oversized surge protectors that usually are offline.
Also make sure it's rated for the load. My UPS went "lol nope" when I lost power while I was gaming, because I had forgotten about it when I went from single mid-range GPU to SLI top-end GPU...
Some of the "newer" online models will have a green mode, which although double conversion green mode introduces a transfer time (~2ms, basically relay lift time).
Backups are also square wave, or stepped sine Smart are Sinewave output.
This is specifically APC.
Also remember there are tare losses (copper / magnetizing and conversions), so a UPS just sitting there can use up to 30W.
Back-UPS stepped sine wave has about 3 square steps above or below zero, so it's a very rough approximation of a sine wave. Adequate for many kinds of equipment, but ugly on a scope.
There is, or used to be, a pin on the Smart-UPS line that you could tie high (or low?), to tell the device to operate in "Online" mode (inverter always active, line power continuously tending the battery). APC also sold it preconfigured that way as a separate, more expensive, product. :)
There's also a trick to jump-starting an unplugged-but-charged Smart-UPS. I've used this for several hours of remote power in inconvenient locations.
Quality devices, but test them regularly. The switch from idle to high-draw can be violent, and it takes out weak components. Similar to tungsten lightbulb filaments.
I'm most familiar with the APC line because that is what I use.
I'm not sure I buy that. The motherboard/GPU backplane VRMs alone should be able to buffer 20ms on these devices. Would love to see a test of this kind of thing.
And this is why professional servers have two, redundant, power supplies.
If you had two power supplies, the UPS would fail, but you'd continue to function off of mains voltage.
UPS can fail. Mains can fail. Two power supplies hooked up two both powering your server redundantly means you'll only fail if both fail simultaneously.
This will probably occur at a different time than when the UPS wears out, so it could be beneficial to keep them separate so you can replace the surge protector only.
It's just that consumer home UPSes are almost scam level products designed to profit from naive people. They are both not improving things for an average user and killing batteries after like a year of operation (lead acid batteries can last for like a decade when not subjected to those UPS chargers).
Ouch. Does anyone know where the author lives that a couple of times a year is considered "infrequent?"
Decades ago I lived in a rural area of America where we'd have power outages two or three times a year because of storms. But I've lived in more than a dozen cities since then, and the power has only gone out twice. Once about a decade ago during a tornado, and then again last year when the power company replaced a transformer across the street.
I ask because it's my impression that electric service has gotten much more reliable over the years. There are still access issues where I live now (believe it or not, thousands of Americans don't have access to the electric grid at home), but reliability seems improved. Am I living in a bubble when it comes to electricity?
Edit: My UPS is a line-interactive rack-mount APC unit, which as I understand things don't actually contain a power supply, so everything runs from the battery voltage and is supported by the charger in mains operation, but without a battery inserted the charger doesn't work, because its control circuit is powered by the battery.
Storms during the rainy season usually means i experience cuts. Usually 1-2 times a month for 3-8 hours ea from May-Nov
Hurricanes are usually expected to be 3+ days at minimum of outage. Most recent ones were closer to 14 days.
I have a couple gennys, and interlock kits with 30a hookups on my house during these times. I can run 1 genny on the whole house, or even 2 since i have a sub-panel with another interlock kit.
The only thing i CANT run is my AC. But we have 2-3 window units/portables to get by.
FWIW running generators for days at a time is NOT cheap. on my setup its about 20-30 bucks a day in fuel alone.
Even with my roof (which is east/west facing) the cost to just install panels would be 20-30k. And my city has a 1:1 buy back...And even assuming i sell them back power it would take a decade to recoup the costs...and by then the panels would be less efficient. And this is with buying 95% panels.
but then during an outage....
So you need something to store it... Short of rednecking a series of deep cycle batteries the Tesla Powerwalls are really the only big option.
And those arent cheap either. To run my house for 2-3 days would be in another 10-30k
I can do all of that with 2x7kw gennys and some ancillaries. With a 14-17kw i could probably run my AC (which have hard start caps on them) Even at 30 bucks a day in gas im still well south of even 10k.
Next step up would probably be a active standby generator with transfer switch and giant Propane tank to fuel it. That would be about 10-15k once permitting and all is done. And that adds complexity and cost. But it would mean i dont have to manually flip breakers and cutover. (as it is, my computers are just set to talk to a NUT server, and shutdown in the event i dont cutover within 10 minutes.)
And during storms supply chains are strained. Its hard to source diesel and LP and sometimes even petrol (in fact i warned a previous company of this, and we almost had to shutdown due to fuel levels). And you have to maintain a contract with LP providers and usually rent a tank from them.
With petrol and my little standby units i can source it myself (3-4 am is the best time during runs on gas), even siphon from one of our cars.
I may spring for a full kit standby one day. But for me, it costs about 2k in generators. Another 1k to have an electrician install the interlock kits. And i can service/manage the gennys myself. swap them out while i do maintenance during extended runs etc.
https://en.wikipedia.org/wiki/Solar_cell_efficiency#/media/F...
So he sized the number of panels to offset 95% of his usage is my guess.
They used to sell them last time i was looking into it, which was about 3 years ago.
The quality of the residential wiring in the original half of the house was not friendly to my hard disks. I lost four in rapid succession. Got a newer UPS, line interactive, had better luck after that until we moved out of state.
(That month of disk meltdown made me love ZFS. Didn’t lose any data.)
What type/size of generator are you running? Might be worth taking a look at an inverter unit next time; they're much quieter and sip gas. Can't stress the quiet part enough - you can hold a conversation a few feet away easily.
Assuming you need 240v since you mentioned you have an interlock kit; Champion makes a 5,000 watt model for $1K. Will go for 12+ hours on 4 gallons of gas at 25% load. https://www.electricgeneratorsdirect.com/Champion-100519-Por...
If you can get by with a single 5-8K BTU window unit, a WEN 56200i/Harbor Freight Predator 2000 (I own one) will last for 8-10 hours on a single gallon, keeping both the window AC and kitchen fridge running. They're a clone of the Honda EU2000i/Yamaha EF2000is, and I've found it to be just as reliable. https://wenproducts.com/products/wen-2000-watt-inverter-gene... & https://www.harborfreight.com/2000-watt-super-quiet-inverter...
I currently have a 7kw (8750 surge) and a 6kw unit. they are dewalt gennys with Honda engines. I really only would trust a B&S, Honda or Kohler engine i feel like.
I think my next purchase will either be a bigger inverter or just a 14-17kw contractor genny and switch to double conversion Battery Backups.
Not quite sure. But what i have does work too. Just that my current generator is approaching a decent number of hours (400-500).
I've lived in many different states and cities, and "it varies" is about the best you can say. Areas I would expect to be relatively problem free, end up being some of the worst offenders (looking at you, El Paso County, Colorado).
[0] https://en.m.wikipedia.org/wiki/Arc-fault_circuit_interrupte...
I had AFCI trip on my fridge after renovating my kitchen while we were away for a weekend and ruined the brand new floors when the ice box melted. So infuriating.
I’ve since pulled the AFCI’s out of all mission critical circuits and sold them on eBay.
Just an absolutely disgusting example of regulatory capture.
I guess not having earthquakes, floods or hurricanes helps.
Where I lived in Sweden, 30% of the network was already underground, but after a large storm in 2005 caused long-lasting power outages, they invested heavily and are now up to 70% underground.
Your electricity service may not be as reliable as you think - some utility power can be quite "dirty" with high levels of distortion. There's a lot of infrastructure between you and the power company that can cause problems. A good UPS will not only protect against outages, but will filter/condition utility power.
I live in Chicago, in an older apartment building. I haven't had a full on outage in the past few years, but during major storms I have seen some flickering and other issues. I just looked at the readout on the UPS, and so far it's kicked in 5 times this year. I have a pretty sizable NAS (40TB) and a beefy workstation - the $200 I spent on an APC BR1500MS is well worth the protection.
Safety is another consideration. My parents live on their hobby farm a few hours away - they are the only house at the end of a mile long road with older power lines. I put a similar UPS to keep their alarm/phone/network online. If there's a medical alarm, fire, carbon monoxide, water leak, etc the alarm can still alert the monitoring company, their internet/VoIP landline can fail over to cellular, and we can get alerted that something is up. Again, the peace of mind alone is well worth the $200 (plus $100 for a battery every few years).
In Southern California, we rarely get rain, let alone major storms and we still lose power 1-2 times/year.
On the other hand, I lived in a number of areas across Canada and power outages were quite rare despite the harsher climate.
PSEG...has their work cut out for them.
PSEG spammed us with all these boasts about how they were prepared.
They weren't.
In other cities I've lived in power cuts were a once every two or three years thing and you could still drive on the roads when it rained.
Another thing to be aware of, if your PC has a PFC power supply, is that it might reboot or have undefined behavior on event of the switchover to battery, unless the UPS is designed to avoid this.
https://www.cyberpowersystems.com/products/ups/pfc-sinewave/
(Why yes, I live in California, land of multi-day power outages. Why do you ask?)
Generally the rack mount models, such as CyberPower OR1500PFCRT2U PFC Sinewave UPS System, 1500VA/1050W, 8 Outlets, AVR, 2U or CyberPower OR2200PFCRT2U PFC Sinewave UPS System, 2000VA/1540W, 8 Outlets, AVR, 2U.
Very happy, and stopped having to replace sensitive flat screens in bad power zip codes (trees vs. lines every other storm, such as for the last week after Isaias).
Although I'm a bit "old school" and generally prefer serial (RS-232) ports, this is one case where the change to USB really was a huge improvement!
APC ships crap batteries and they often do not survive the warranty or die shortly later.
https://perspectives.mvdirona.com/2013/02/the-power-failure-...
Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.
Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators.
If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should.
Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.
In the end you WILL test what happens when the hospital plunges into darkness. Would you prefer the first test to be a a bright sunny day when all the staff is ready for something (and you can turn the utility power back on quickly if something fails), or when someone with a backhoe/chainsaw has an accident with your power feed. The latter will happen, I don't know how or when, but at some point in your future trees will take down power lines, backhoes will cut something, major weather events will take out power for days, maybe a large blackout...
You should have course test everything separately often. However if you wouldn't let a random electrician (electrician only because they will know how to not kill themselves) turn off your utility connection with only a day of warning, your system isn't trusted to handle real events which are typically much worse than a clean turn off the switch.
FYI where I’ve worked the tiers of loads are metered individually and picked up in order of priority and with a check to verify the combined capacity of the generators which are online is great enough to pick up the load. Likewise loads are shed in order if there isn’t enough generation such as one generator tripping.
I don’t really see that much difference between opening a breaker and bad quality power. It is the same sensing and logic in both cases: is the magnitude and frequency of the phase to phase voltages in the acceptable range or not?
Wow!
Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster
Additionally, there's the Mach Cone. The shock wave off the nose forms a cone shape, with the angle determined by the speed of the aircraft. From the nose to the wingtip forms an angle of about 17.5 degrees, which corresponds well with the max speed from the CIT of Mach 3.3.
That both methods of determining max speed agree shouldn't be surprising. Skunk Works was filled with good engineers.
The airframe, however, will get bent from temperature and you better hope the missile cooling loops don't fail.
The power isn't dialed down at lower speeds in order to provide thrust for maneuvering and climb.
The boost controller has a "scramble" button which overrides the preset boost targets and allows you to run a higher (and unsafe) boost level as long as the button is pressed.
This is useful when you're in the middle of a $10,000 race and you're losing by a few feet.
The "scramble" button I'm talking about is for "boost", which is the amount of positive air pressure inside the intake manifold.
Both "boost" and nitrous can dramatically increase power by essentially getting more oxygen into the combustion chamber, but nitrous is a much more dramatic effect.
This mode is pretty much reserved for go around or post V1 takeoff when one engine on twin engined airliner had failed.
Looks like there's a drop in replacement for Halon.
Older systems, before Halon was invented, are typically CO2 or Argon. They require a lot more agent to accomplish the job, but they're very simple.
I worked briefly in a CO2 datacenter. It dated from the mainframe era, and had some gorgeous mementos on the walls. There were SCBA packs in the hallway outside, and everyone got a brief safety lecture before being allowed into the facility. The point I remember was basically "The alarm will sound intermittently for twenty seconds, then go solid for five, before discharge. If it goes off, do not stop to save your files, just run for the door. If you're not outside yet when the solid tone sounds, hold your breath and run on what's in your lungs."
Apparently the CO2 is considerably nastier than Halon to get a lung-full of...
- The Stratus supposedly had fault tolerant duplexing of boards - pull one out and the other keeps working. A famous demo where local consultant pulls out the only board which wasn't duplicated!
- The machine room was also fire proofed with Halon gas (if I recall correctly), but due to security concerns the door was locked from inside by operators when present (luckily as programmers we weren't expected to work inside with doors locked!). Luckily also, it was my first exposure to "paired programming"/"paired operating" as in there were invariably 2 of them.
- A couple of the operators became too interested in how to work things, and wrote a quite comprehensive manual. They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves".
- the whole headquarters building was in a guarded compound, where ultimate protection against "red brigade terrorists" (https://en.wikipedia.org/wiki/Red_Brigades) were the armed guards. We used to meet them in the ground floor cafe every mid-morning, with a revolver on their hip, while they drank an espresso with brandy chaser. Really inspired confidence that they would be there to protect us should the need arise!
And this is why we cannot have nice IT security practices.
In a datacenter though your bets are usually much lower. Nobody is going to die, or lose many millions, if some of your equipment shuts down to prevent it from being damaged, catching a fire, etc.
There are though high-stake situations where you want exactly that: spend the entire amount of the resources of certain hardware to prevent a loss of life, or of untold millions, when you are powering a surgery chamber, or a large stock trading operation.
People who realize that do over-provision and pay top dollar for that when they can afford that. I remember that when a major fire occurred in one of the skyscrapers in the financial district of NYC, traders of a particular financial company were evacuated with their laptops into helicopters on the roof, and ferried to a spare office across Hudson river, in NJ. To minimize the impact of that, they connected to the corporate network via their phones as hotspots, and kept trading while airborne.
Of course one cannot hope to pull such an operation off without extensive preparation and likely regular drills.
Not preparing to a black swan event during a high-stakes event, like translation of a Superbowl match, sounds like either not having enough paranoia which is professionally required, or, more likely, as a cost-cutting after a wrong assessment of risks.
(2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.
Day traders are expensive? I thought a lot of those folks keep what they kill, and when they screw up they don't get to eat. "Coffee for closers," etc.
If they trade for a large corporation, with the leverage provided by that corporation, it's a whole another ballgame.
> According to Allied Ordnance Publication AOP-38-3,[1] a NATO publication, a battleshort is "The capability to bypass certain safety features in a system to ensure completion of the mission without interruption due to the safety feature." It also says, "Examples of bypassed safety features are circuit overload protection, and protection against overheating".
> For example, the electrical drives to elevate and traverse the guns of a combat warship may have "battleshort" fuses, which are simply copper bars of the correct size to fit the fuse holders, as failure to return fire in a combat situation is a greater threat to the ship and crew than damaging or overheating the electrical motors.
> Battleshorts have been used in some non-combat situations as well, including the Firing Room/Mission Control spaces at NASA during the manned Apollo missions — specifically the Moon landings.
As a side note, with great apology to those who may have lost their lives due to the consequences of real-world battleshorts - this is actually a really interesting analogy for the "war room" scenarios we find ourselves in at startups. Risking damage to morale and productivity can be acceptable if the alternative is irrecoverable loss of the startup's reputation. But this also can't be a sustained state of affairs - these types of procedures are meant for a battle, and risk compounds if you don't return to a steady state. Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
Thank you for this, I'll be updating our documentation to include these concepts first thing tomorrow.
... And 6 months later marketing people will have commandeered it, without regard to the original meaning, and they’ll be asking people to battleshort so they can really be agile and lean in on providing air cover via the updated sales deck.
We're now sharpening pitchforks to use on the next salestrooper that comes close to the SalesBell™
/s
Doubly so because I used to be in the US military but now live in a different country, and it's crazy to see how those terms and acronyms infect a lot of US business culture.
A decade ago I would have been fine, even glad, that I could wrap myself in verbiage that would jive with poor career choices I made at 18. Now it just feels incongruous or jingoistic.
The owner of the datacenter's aversion to downtime is not going to be a concern to the building inspector or the fire marshal.
At one of my previous workplaces I learned a bit about our local firecode in relation to electrical code. Mostly that firecode supersedes electrical code. The fire alarm is running of a 230V line (standard) without grounding (non-standard) via a special line from the upstream power transformer that has higher amperage tolerance.
All the fire detectors are powered and wired from this at 40-70V, no ground or if there is a ground, it's grounded at the control panel of the fire alarm system.
In the event of a fire, the entire system is rated about 10X over it's standard ratings. The wires that are allowed to handle 1 Amp before are now allowed to go to 10 Amps, just in case there is a short this might burn it out and allow other equipment to operate.
I've seen some systems that are "optionally fused", they have an internal relais that's held by external power and if the fused lines fail, it automatically switches to unfused power, where the alarm system can evaluate the condition and decide to go back to the fused line if power returns. That's on top of having it's own backup batteries. Some of the more modern systems have watchdogs systems built in so that in case the fire alarm computer is on fire, it won't switch to the unfused line once the computer is no longer operational.
At to amperage ratings, don't you just mean that they have a 10x safety factor? Probably using current limiting supplies that can handle a short circuit. Fire circuits (and the feed from the main panel) should use fire rated cabling -- generally mineral insulated copper sheathed cables. Most of the standards (NFPA 75/76) are moving away from being prescriptive to being risk or performance based though, so it is possible they used normal conduit if the overall risk was small.
Incidentally, hospitals also make extensive use of isolation transformers for wet areas (like emergency / ICU), to provide a local point for resetting breakers (electronically), identifying which point is tripping, etc, see IEEE 602-2007. Much better filtering for medical equipment too. You also tend to see positive retention plugs (in the US, green dot UL 817).
The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline).
Test components individually - at the factory and in the field - and test systems end-to-end.
Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope.
When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system?
Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails".
If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?
http://resistive-loadbank.sell.everychina.com/p-109417602-35... is the first example I pulled up, and there are bigger ones.
I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.
Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas.
After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power.
Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back.
When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank.
Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
Once a quarter we'd do a live load transfer, and watch all the PDUs and UPS systems do their thing. Those could be stressful...
You could just, you know, pull the plug and see what happens?
Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online.
I never did get a chance to see the generator room.
e: Another personal anecdote that is more relevant to the topic.
I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.
The blog where they posted regular updates is still up; here's the first post related to the storm: https://interdictor.livejournal.com/2005/08/27/
"Hmm. This could actually be a nasty storm."
Very good read!
Run a combined-cycle natural gas generator 24/7, and use the grid as your backup. Gets good efficiency. Runs loops on the coolant, oil and exhaust for water/building heat, and can switch over instantly in the event of failure.
https://www.solidwastemag.com/feature/talking-bout-cogenerat...
Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?
Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand.
But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.)
If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage.
But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
The customer was running the generators right at the maximum capacity, and we'd see really, really odd things on the power lines. Some times a surge like you mention, sometimes changes to the frequency, sometimes changes to the voltage. We discovered all sorts of weird brownout conditions in our devices.
In our small data centre we have a board with five lights and a key. You turn the key and the mains is cut off. The lights give the status. The UPSs beep for a few seconds "on battery", the genny fires up in the boiler room and when it stabilises, it takes the load. We leave it running for 5-15 minutes, go out and check the fuel levels and top up accordingly. It's diesel so safe to hot refuel. When done, turn the key back and then the UPSs take the load again and then hand back to the mains. The genny runs for another 15 minutes and then shuts down. That should avoid flip flopping.
You then fill in the test form on our wiki and job done. We do it weekly to fortnightly. An ESP8266 based thingie reports status back to the central monitoring (GPIOs and ESPHome -> Home Assistant)
If my batteries die, I'll just replace them with non-battery backup strips. Not worth the trouble for me living in a city where power goes out maybe once per year on average for 10 minutes, and I don't have servers that need to stay online anymore.
But the main point in the article is worthy: UPS manufacturers have been skimping somewhat on functionality, features and quality over the past few years...
I still have a few APC Smart UPS's bought in 1999-2000 running in my house; at the same time I had a few cheapies that broke after a few years, sometimes in spectacular ways (electronic exploded melting part of the battery case); I will not name brands, it is not worth.
Since 2000 I changed cases full of batteries, usually every 2-3 years. The good thing with the APC Smart series is that it works even if it gives an alert that the battery is bad, but you will get a few minutes of power instead of 30-60 minutes (my server & NAS is very low power, 5% of the UPS capacity). I also set the server to shutdown when the battery is reaching 50%, so I never deplete the batteries. At the same time, the UPS on the water pump is usually running on batteries as much as it can, so I need to change these batteries after 2 years. I have both the server and the water pump in the basement, the temperature is around 16-20 degrees Celsius (winter-summer), when I was keeping the server and the UPS in the house the battery life was at half (~18 months). The heat and the deep discharge cycles are effectively killing batteries.
I had the same type of battery on my motorcycles, one lived about 8 years and a second one 6 years, but in very different conditions: it was warmer, but almost no discharges.
I've used UPSes that discharge batteries no lower than 10.5 (no deep discharge) and only once in a couple of years and that charge them to and keep them at 13.6 volts under stable room temperature all year. All batteries lost most of the capacity after 3 years, two almost all of it, one had like 40% left. In five years only that one was still surviving short minutes outages, while initially was able to do a few hours. For comparison, I had one battery just laying around for 6 years in the same room without touching it and it lost only like 30%.
I found this forum posting while searching for how to maximize the runtime on my APC UPS: https://forums.overclockers.com.au/threads/hacking-the-newer...
Maybe APC really wants to drive users of its lower tier (and cheaper) UPSes mad with the beeps and force them to buy the more expensive Smart-UPS range that comes with a built-in option to turn off audible alarms.
Well, audible alarms do have a place in such systems to inform the person that the connected devices or systems are running on a limited battery capacity. But forcing that at all times is a big annoyance.
I meant to tear it apart and remove the beeper after that but never got around to it before the battery made it useless anyway. TBF it was a few years old at that point so the battery giving up wasn't unexpected.
https://www.amazon.com/gp/product/B01FWAZEIU/
The beeping didn't really bother me but it does frighten one of my dogs.
Agreed. The cyberpower devices do let you "silence" the alarm, but even with the alarm "silenced" the UPS will begin rapid beeping as the battery nears depletion.