I think that people understand the level of engineering and technology that goes into batteries and their designs these days. There’s a ton of talent, a ton of continuous innovation, it in the end it too often gets relegated to a black box for energy in and energy out. Which is in some ways a testament to the quality of the engineering.
> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive.
A short in an offline device? That seems weird.
> A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.
So a new pack was installed and connected but not monitored? What is the rationale behind that? Failures shouldn’t happen to new devices?
An extract from the summary page suggests a few things went wrong, but it seems like the fire was caused by liquid and electricity mixing:
> ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment, which consequently damaged two Megapacks.
> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive.
> A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.
Why would Tesla have the "worlds best" engineering? I don't think I have ever seen anyone ever claim that except for perhaps Tesla PR representatives. How would you even measure "best" in such a field? Cheapest? Best quality? Best value-for-money? Safest? Highest power density?
I am very surprised in this conclusion... Normally if you have two high voltage wires being immersed in water, a large current flows, vaporising the water, and then the current stops when all the water has been evaporated.
For the current to flow for long enough to heat the battery packs to cause a fire, there must have been a jet of water flowing for a long period of time. And no fusible links between that place and the battery.
It really seems like either very bad luck, or bad system design, or both.
Sure you can do it, but it won't be reliable or cost effective.
Passive water cooling in natural water tends to lead to seaweed and all kinds of sea creatures blocking up all heat exchange surfaces.
In the case of a megapack, cooling isn't critical to safety - as long as you're happy to stop using a battery when too hot, it won't get hotter. The reason to cool is is so you can keep using the battery.
The article puts it this way: "ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment." Nothing here implies that the damaging heat generation occurred at the location of the short circuit. It could have been at any point in the new current path created by the short circuit, depending on its resistance and the share of the overload current it carried.
You folks are acting as if some huge crime against humanity happened here. In reality nobody got hurt, the company who messed up will pay to put it right. They already indicated (in the linked report) that they learned from the incident and changed multiple things in their systems and procedures as a result.
It's a live (active) power source even if it's being commissioned. I don't think it's a crime against humanity but it sure created pollution and waste.
> It wasn’t operational, it was being comissioned.
You're telling me that you don't see a problem with the idea of starting up a bleeding edge system intended to store incredible amounts of energy without the monitoring systems connected?
In general if you don't have a way to monitor an industrial-scale system you go in to your fail-safe state.
Each of these Megapacks can contain 1.5% of the energy contained in a petrol station. :) Let’s not get ahead of ourselves, these are realy nice and big batteries. That being said the energies stored in them are far from incredible.
These boxes were by all accounts manufactured in Nevada. They have been moved all the way to Victoria, lowered to the ground and hooked to the site’s wiring. They were not “charged” yet. The next step as part of the comissioning was to hook them up to the monitoring system before they are checked out.
No matter when you hook them up to be monitored there is a step just before it when they are not monitored yet.
> In general if you don't have a way to monitor an industrial-scale system you go in to your fail-safe state.
That is indeed a great idea. And if you read the report you can see that after they hooked up each box they put them manually into “maintenance” mode. This mode de-energises all systems in the box. It totaly makes sense that they thought this procedure is the correct one to reach the fail-safe state you mention. Turns out they were wrong. They learned this now and changed their procedures.
Tesla’s model is they both sell the Megapack system and then orchestrate it over its lifetime using their Autobidder platform. The owner is responsible for financing, integration with on site generation, and working with the local grid operator.
> "A new battery module isolation loss alarm has been added."
This is the worst fault possible in an EV 101 scenario right here (touch metal -> instant death), so I'm surprised that with all Tesla's experience, they didn't consider an isolation leak as a serious fault. And I'm surprised that they allow the battery monitoring system to be disabled during servicing, especially the IMD (insulation monitoring device).
I am constantly surprised how many systems have no isolation when it would be easy and cheap to add.
For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life.
Considering solar inverters already uses high frequency switching inside the per-string MPPT boost converter, it would have cost mere cents to put two windings on that inductor rather than one and get isolation with no downside.
The cynic in me says manufacturers don't add safety features not required by law, even if zero cost, because in the future laws might be updated to require that safety feature, preventing reuse of hardware already sold, which is good for business.
> For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life.
This is very yikes. I just installed such a system.
Do you know any way to test if that's a problem? And is it possible to fix without opening the inverter?
It's normally set that high by design because the solar panels have substantial capacitance to ground (they are large flat conductors sat just a few cm off the ground after all). The capacitance to ground wouldn't be an issue if not for the fact the inverter is non-isolated. That means current can flow to/from the grid to charge and discharge that capacitor, which is indistinguishable from AC leakage.
Some inverters will allow you to lower the leakage threshold (usually with settings in a service menu), but then you're limited to just attaching 3 or so panels or getting false isolation trips.
I have heard people say "that can't be right, because a car battery can source 100+ amps and touching those terminals doesn't kill me!" The reason is because you have a high enough impedance that 100mA doesn't flow from a 12V source through your body.
"It's not the voltage that kills you, it's the amps!"
Wouldn't power (W) (or joules) be a better way to phrase it?
Because as you said, you can't have 100mA flowing through your heart unless you ALSO have stiff enough source to actually supply that current - which implies a high compliance voltage, can't escape ohms law.
No, it's not the power, either, as low resistant paths can be extremely dangerous (such as wet skin). Really, it's a 2 dimensional space that makes for a less catchy aphorism. "It's not the voltage that kills you, it's the combination of voltage and conductivity".
Honestly I kinda hate that saying. It's like "it's not the fall that kills you, it's the sudden decelleration on impact." For most situations people encounter, it's high voltage sources with more than enough power which are dangerous.
GP mentioned impedance, not resistance. You won't feel anything touching the 12/24V terminals in your car because they're DC. Your skin has a high resistance, but when faced with AC, has a low impedance. This makes AC much more dangerous, as you conduct 120V AC much better than 120V DC.
Even 50mA will likely kill you, as outlined on the "death graph" as we electricians call it. This is independent of voltage.
Blue is harmless and imperceptible, green is harmless, yellow is harmful, red is fibrillation.
Edit: This illustrates why 30mA (or less) RCD protection is common (and required) in various jurisdictions, depending on the kind of circuits you're serving.
Right, and in particular, "Type B" RCD protection is required for solar inverters in various jurisdictions. These provide protection against DC fault currents.
I can't answer that question, as I'm only an electrician, not an electrical or RF engineer.
That said, we're referring to power delivery here, and the highest frequency I'm aware of in this space is the 400 Hz power distribution systems used on warships.
Embarrassingly, I have managed to shock myself several times during my life on 240V mains. Once quite recently while working on a light fitting at a friend's flat and not realising that someone had flipped the circuit back on at the circuit breaker!
That shock was certainly quite unpleasant, but despite touching my sweaty skin against live wires, it seems to have been far from fatal.
Surely the dangerous situations are live-to-earth shocks, which could potentially run across your whole body (rather than just a hand or finger, as in my case). But that's why we have RCD/GFCI protection, right? Those will trip with tiny fault currents, certainly much less than 300 mA.
Secondly, aren't "Type B" RCDs required on solar inverters? Those will detect and trip on DC fault currents as low as 30 mA, not 300!
> Embarrassingly, I have managed to shock myself several times during my life on 240V mains. Once quite recently while working on a light fitting at a friend's flat and not realising that someone had flipped the circuit back on at the circuit breaker!
Next time, I suggest using a breaker lockout [0] in conjunction with the local disconnecting means (light switch, which you can also lock out! [1]) to do it safely :)
No you can't! Ceiling light fittings (in the UK, at least) are often always live regardless of the switch setting. This is because they are typically wired in a "loop" arrangement with separate loop and switch cables.
Well, the rest of the world outside of the UK, Ireland, and Hong Kong uses a safer method of electrical circuiting, one that wasn’t invented to deal with post-war material shortages.
> A domestic power supply voltage (110 or 230 V), 50 or 60 Hz alternating current (AC) through the chest for a fraction of a second may induce ventricular fibrillation at currents as low as 30 milliamperes (mA).With direct current (DC), 300 to 500 mA is required.
Galvanic isolation is useful because it means that you have to touch 2 specific points on the affected unit simultaneously to die, rather than 1 point. Galvanically isolated systems can detect leaks on both the positive and negative sides by using a dual channel IMD. Any insulation issue is bad, but this method of detection can usually detect it before it's able to instantly kill someone per-se. Electric vehicles are an example of a galvanically isolated system (from earth) with dual channel IMD, hence you can have massive leaks such as 8-20mA before it becomes a safety issue.
Galvanic isolation probably wouldn't have prevented this fire, it sounds like the cells just shorted together. They would have had to short over several cells, since a non-trivial voltage is required to break down coolant, so having maintenance contactors to break the pack up into 20V or so segments may have prevented this fire.
Using a mineral oil type coolant that is not electrically conductive (such as used in Tesla vehicles) would have also helped to prevent this fire.
The batteries should be galvanically isolated from the casing so that if liquid causes a current flow from any point in the pack, then the maintenance contactors can be opened and the opposite side current path for the leak is interrupted. We can only assume that they are already (galvanically isolated) since anything else would be bizarre.
I spent a bit working on various power supplies and sadly isolation is non-trivial nor cheap. Adding isolation (transformer vs inductor) requires changing control signals, control mechanism, size. It costs far more then mere cents, and loses some efficiency.
Reading this it’s a classic systems failure: multiple defensive barriers gone wrong through poor safety planning. It’s reminiscent of the Victorian ESSO gas plant accident https://en.wikipedia.org/wiki/Esso_Longford_gas_explosion
- one involves a gas utility the other an electric one
- one has killed and hurt workers while the other did not killed nor injured anyone.
- one happened after several years of operation, the other happened during installation.
- one left the state without energy service for 20 days, the other did not affect service at all.
Don’t get me wrong. I love reading about how complex things go wrong, one of my favourite pastimes. So I thank you for your link. It just doesn’t seem that similar to me. Not sure you would be citing the same case for example have they not happened in the same state.
If I have to choose a failure which the Big Battery fire reminds me of I would rather choose the Florida International University pedestrian bridge collapse. Why? It seems these batteries had adequate safety functions and monitoring planned in for their operation. Just these systems were not activated during the installation in the right order. Similarly with the bridge it seems they did calculations with the full bridge in place, but nobody seems to have checked if the intermediate steps during the construction will stay up too.
Another similarility (to my untrained eyes) is the aim to install a system fast. The bridge project wanted to be a flagship for accelerated bridge building, and the big selling point of these Megapack installations is how fast they can be comissioned. With the bridge project we know that this demand for speediness was a contributing factor. Was it maybe also with the Big Battery fire too?
>In what way is it reminding you of that incident?
It's just in group signaling. "Hey this reminds me of <vaguely related thing>" is a more polite way of saying "look I know about X, I'm like you, gimme dat virtue points"
You see this on literally every virtue points (i.e. up-vote) based form of social media that's big enough that any given participant can blend in with the crowd.
The behavior is endemic these days you can't even post a picture of a rotten and falling down deck on the internet these days without some jerk derailing everything by dropping a link to the wikipedia page on the hyatt regency and then several more riding their coattails with quotes copied from the page.
He probably doesn't even consciously realize he's doing it.
for people who might say 'oh, but these batteries are so dangerous!'
How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year?
I imagine the largest insurance companies have reasonably good data on this as a cause of total-loss of a vehicle.
How many times have you seen in person, or seen a photo of a burned out RV that somebody pushed too hard up a mountain pass without keeping an eye on the engine temperature?
The point was that plenty of other energy dense things with radiator-based liquid cooling loops lack proper monitoring systems.
Not excusing Tesla but the actual number of heat producing things that catch on fire every year (in general) because of cooling pump/cooling loop failure is quite a lot. In many categories of equipment.
You have a gross misconception of the relative temperatures of components in internal combustion engines, and you're using a faulty conclusion based on the same misunderstanding as supposed supporting evidence.
I assure you an RV or other gas/diesel vehicle does NOT suddenly go "pop!" and produce an engine fire when the needle reaches the right hand side of the temperature gauge.
The temperature gauge warns you when the coolant is at risk of boiling; it's not a warning that the engine is about to spontaneously combust.
right, that's why every burned out at the roadside RV I've seen in my lifetime (probably more than a dozen now) was on the upward slope of a high mountain pass, or at the peak...
one of the points I was trying to make is that people don't maintain liquid cooling loops in general, and that's one of the more common instances of it.
do you live somewhere flat?
go ask the towtruck drivers who work the coquihalla highway in BC how many vehicle fires they see every year, and where those vehicle fires occur.
84 comments
[ 1.8 ms ] story [ 147 ms ] threadIt's a battery pack, not a stash of U235 above the critical mass. Every recycling facility is more dangerous.
"The affected Megapacks failed safely despite total loss."
A short in an offline device? That seems weird.
> A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.
So a new pack was installed and connected but not monitored? What is the rationale behind that? Failures shouldn’t happen to new devices?
Any time a house is built, there's some time between the ceiling going up and the fire alarm installation being completed.
Nothing wrong with that - if it's done properly and carefully.
> ESV found a Megapack cooling system leak caused a short circuit resulting in overheating that led to a fire in a nearby battery compartment, which consequently damaged two Megapacks.
> There were further contributory factors with the Megapack in question being switched into an off-line service mode, resulting in the protection systems being inactive.
> A 24-hour delay in connecting the batteries to the supervisory control and data acquisition (SCADA) system also meant there was no active monitoring of the Megapack alarms.
And the cooling system should be continuously monitored for faults. Not just inspected after filling.
This isn't some weekend hackathon contest, Tesla is supposed to have the worlds best engineering.
https://en.wikipedia.org/wiki/Tesla_Megapack
I would imagine this would be really fucking hard.
Most larger, more expensive engineering units don't have this.
I'm open to the idea, if it exists or there are similar setups then that would show it's possible.
Else I'd guess this is like creating Facebook in a weekend. Easy to say. Regulations, security, false positives, unique software.
For the current to flow for long enough to heat the battery packs to cause a fire, there must have been a jet of water flowing for a long period of time. And no fusible links between that place and the battery.
It really seems like either very bad luck, or bad system design, or both.
Of course issues with sweet vs salt water
Passive water cooling in natural water tends to lead to seaweed and all kinds of sea creatures blocking up all heat exchange surfaces.
In the case of a megapack, cooling isn't critical to safety - as long as you're happy to stop using a battery when too hot, it won't get hotter. The reason to cool is is so you can keep using the battery.
What the actual... how is something like this even allowed to be operational without monitoring?
You folks are acting as if some huge crime against humanity happened here. In reality nobody got hurt, the company who messed up will pay to put it right. They already indicated (in the linked report) that they learned from the incident and changed multiple things in their systems and procedures as a result.
You're telling me that you don't see a problem with the idea of starting up a bleeding edge system intended to store incredible amounts of energy without the monitoring systems connected?
In general if you don't have a way to monitor an industrial-scale system you go in to your fail-safe state.
Each of these Megapacks can contain 1.5% of the energy contained in a petrol station. :) Let’s not get ahead of ourselves, these are realy nice and big batteries. That being said the energies stored in them are far from incredible.
These boxes were by all accounts manufactured in Nevada. They have been moved all the way to Victoria, lowered to the ground and hooked to the site’s wiring. They were not “charged” yet. The next step as part of the comissioning was to hook them up to the monitoring system before they are checked out.
No matter when you hook them up to be monitored there is a step just before it when they are not monitored yet.
> In general if you don't have a way to monitor an industrial-scale system you go in to your fail-safe state.
That is indeed a great idea. And if you read the report you can see that after they hooked up each box they put them manually into “maintenance” mode. This mode de-energises all systems in the box. It totaly makes sense that they thought this procedure is the correct one to reach the fail-safe state you mention. Turns out they were wrong. They learned this now and changed their procedures.
Interesting that even after installation Tesla are still involved in operationally managing the megapacks.
This is the worst fault possible in an EV 101 scenario right here (touch metal -> instant death), so I'm surprised that with all Tesla's experience, they didn't consider an isolation leak as a serious fault. And I'm surprised that they allow the battery monitoring system to be disabled during servicing, especially the IMD (insulation monitoring device).
For example, typical solar inverter systems don't have isolation between the AC and DC sides. That means any chafed wire can be deadly. They do at least have leakage detection, but set at a 300mA level, which probably won't save your life.
Considering solar inverters already uses high frequency switching inside the per-string MPPT boost converter, it would have cost mere cents to put two windings on that inductor rather than one and get isolation with no downside.
The cynic in me says manufacturers don't add safety features not required by law, even if zero cost, because in the future laws might be updated to require that safety feature, preventing reuse of hardware already sold, which is good for business.
This is very yikes. I just installed such a system.
Do you know any way to test if that's a problem? And is it possible to fix without opening the inverter?
Some inverters will allow you to lower the leakage threshold (usually with settings in a service menu), but then you're limited to just attaching 3 or so panels or getting false isolation trips.
https://academic.oup.com/ptj/article-abstract/46/9/968/46379...
I have heard people say "that can't be right, because a car battery can source 100+ amps and touching those terminals doesn't kill me!" The reason is because you have a high enough impedance that 100mA doesn't flow from a 12V source through your body.
Honestly I kinda hate that saying. It's like "it's not the fall that kills you, it's the sudden decelleration on impact." For most situations people encounter, it's high voltage sources with more than enough power which are dangerous.
Even 50mA will likely kill you, as outlined on the "death graph" as we electricians call it. This is independent of voltage.
https://upload.wikimedia.org/wikipedia/commons/7/7f/IEC_TS_6...
Blue is harmless and imperceptible, green is harmless, yellow is harmful, red is fibrillation.
Edit: This illustrates why 30mA (or less) RCD protection is common (and required) in various jurisdictions, depending on the kind of circuits you're serving.
But at higher frequencies, you should experience the skin effect, right?
That said, we're referring to power delivery here, and the highest frequency I'm aware of in this space is the 400 Hz power distribution systems used on warships.
That shock was certainly quite unpleasant, but despite touching my sweaty skin against live wires, it seems to have been far from fatal.
Surely the dangerous situations are live-to-earth shocks, which could potentially run across your whole body (rather than just a hand or finger, as in my case). But that's why we have RCD/GFCI protection, right? Those will trip with tiny fault currents, certainly much less than 300 mA.
Secondly, aren't "Type B" RCDs required on solar inverters? Those will detect and trip on DC fault currents as low as 30 mA, not 300!
https://en.wikipedia.org/wiki/Lockout–tagout
Next time, I suggest using a breaker lockout [0] in conjunction with the local disconnecting means (light switch, which you can also lock out! [1]) to do it safely :)
[0] https://www.walmart.com/ip/MASTER-LOCK-493B-Circuit-Breaker-...
[1] https://www.homedepot.com/p/Ideal-Wall-Switch-Lockout-44-789...
No you can't! Ceiling light fittings (in the UK, at least) are often always live regardless of the switch setting. This is because they are typically wired in a "loop" arrangement with separate loop and switch cables.
Example: https://www.practicaldiy.com/electrics/lighting-wiring/light...
https://en.m.wikipedia.org/wiki/Electrical_injury
Mike Holt is an excellent instructor on the NEC and has quite a few videos that speak about this topic.
https://m.youtube.com/user/MikeHoltNEC
Are you working in EE?
Using a mineral oil type coolant that is not electrically conductive (such as used in Tesla vehicles) would have also helped to prevent this fire.
The batteries should be galvanically isolated from the casing so that if liquid causes a current flow from any point in the pack, then the maintenance contactors can be opened and the opposite side current path for the leak is interrupted. We can only assume that they are already (galvanically isolated) since anything else would be bizarre.
The paralels I can see:
- both involve energy installations.
- both happened in the same state.
The differences I can see:
- one involves a gas utility the other an electric one
- one has killed and hurt workers while the other did not killed nor injured anyone.
- one happened after several years of operation, the other happened during installation.
- one left the state without energy service for 20 days, the other did not affect service at all.
Don’t get me wrong. I love reading about how complex things go wrong, one of my favourite pastimes. So I thank you for your link. It just doesn’t seem that similar to me. Not sure you would be citing the same case for example have they not happened in the same state.
If I have to choose a failure which the Big Battery fire reminds me of I would rather choose the Florida International University pedestrian bridge collapse. Why? It seems these batteries had adequate safety functions and monitoring planned in for their operation. Just these systems were not activated during the installation in the right order. Similarly with the bridge it seems they did calculations with the full bridge in place, but nobody seems to have checked if the intermediate steps during the construction will stay up too.
Another similarility (to my untrained eyes) is the aim to install a system fast. The bridge project wanted to be a flagship for accelerated bridge building, and the big selling point of these Megapack installations is how fast they can be comissioned. With the bridge project we know that this demand for speediness was a contributing factor. Was it maybe also with the Big Battery fire too?
It's just in group signaling. "Hey this reminds me of <vaguely related thing>" is a more polite way of saying "look I know about X, I'm like you, gimme dat virtue points"
You see this on literally every virtue points (i.e. up-vote) based form of social media that's big enough that any given participant can blend in with the crowd.
The behavior is endemic these days you can't even post a picture of a rotten and falling down deck on the internet these days without some jerk derailing everything by dropping a link to the wikipedia page on the hyatt regency and then several more riding their coattails with quotes copied from the page.
He probably doesn't even consciously realize he's doing it.
If you can make battery packs air cooled, that's a big, big weight saver.
How many cooling system leaks (radiator+water pump related plumbing) result in vehicle engine fires per year?
I imagine the largest insurance companies have reasonably good data on this as a cause of total-loss of a vehicle.
How many times have you seen in person, or seen a photo of a burned out RV that somebody pushed too hard up a mountain pass without keeping an eye on the engine temperature?
The fault lies square with Tesla. No amount of RV analogies can dilute this.
Not excusing Tesla but the actual number of heat producing things that catch on fire every year (in general) because of cooling pump/cooling loop failure is quite a lot. In many categories of equipment.
Almost none. Overheating is highly unlikely to cause a fire. Fuel leaks cause fires.
I assure you an RV or other gas/diesel vehicle does NOT suddenly go "pop!" and produce an engine fire when the needle reaches the right hand side of the temperature gauge.
The temperature gauge warns you when the coolant is at risk of boiling; it's not a warning that the engine is about to spontaneously combust.
one of the points I was trying to make is that people don't maintain liquid cooling loops in general, and that's one of the more common instances of it.
do you live somewhere flat?
go ask the towtruck drivers who work the coquihalla highway in BC how many vehicle fires they see every year, and where those vehicle fires occur.
Fuel leaks and/or exhaust system failures, especially on turbocharged vehicles however...