559 comments

[ 0.21 ms ] story [ 328 ms ] thread
The rocket is currently in flames. Staffers at the facility are telling me. Plumes of smoke rising from Launch Complex.
Looks like this may have been the Falcon 9 getting ready to launch AMOS-6 on 9/3. Not a previously flown rocket.
Static Fire for that launch was due today which is probably during which this happened.
Wild-ass speculation in the absence of more information than a tweet and a photo of a pad fire:

SpaceX test-fire the first stage motors before each launch. If this was a test firing that went spectacularly wrong, it's embarrassing -- but there won't have been any human beings within blast range and it's better to fail in test than to fail in flight with a payload on top.

(If it was a catastrophic failure during fueling/de-fueling ops, that's another matter entirely, and far more serious -- and an explosive test failure is serious enough as it is.)

Test fire for the launch of a commercial satellite (AMOS-6), which had been scheduled for a few days hence (and now presumably delayed, along with the rest of their launch schedule, pending further investigation). It might conceivably have been during fueling/de-fueling around the test. It's not yet clear from public reports whether the payload was attached at the time of the test-fire; sometimes they are, sometimes they aren't.

(Note that fueling operations for Falcon 9 got a whole lot trickier when they switched to use of super-cooled fuel and oxidizer, to increase density -- if that stuff gets warmer, its volume increases to more than that of the tank, but it's still liquid, so you can't just vent it off.)

(comment deleted)
> of a commercial satellite (AMOS-6)

According to wikipedia this is related to the AMOS-2 staelite leased by Facebook to provide internet coverage in part of Africa.

An interesting clause in their contract is:

"The parties have agreed to the right to terminate the contract if Amos-6 and the ground gateways in France, Italy and Israel are not ready for service by January 1, 2017"

So this fire could have wider repercussions if SpaceX doesn't get a new launch ready in time. (Unless that same contract also has a clause about SpaceX failure vs IAI failure.)

> but it's still liquid, so you can't just vent it off.

You can vent off liquid to keep the pressure down, but since it's heavier than air it will just add to the fire. Unless maybe they have some sort of piping system connected to the vent to route the extra fuel elsewhere.

Unless they happen to have a backup satellite already built, nothing SpaceX can do is going to accomplish this mission in 2016. There's no chance whatsoever that IAI is going to crank out a new satellite in just 3 months.
All of SpaceX's launches are likely on hold for a few months, anyways.
So does that put the rocket in a situation where its either launch or warm up and explode? I'm adventurous, but I don't want to be strapped to that thing if thats the case.
I'm sure it's not like that. Pressure relief valves aren't exactly rocket science.
You can't vent rocket fuel, the kerosine will ignite from the expansion and the friction in the valve alone.

Venting liquid oxygen is also a really really bad idea.

On the contrary, venting LOX is very common and easily visible in the lead-up to nearly every Falcon 9 launch. Google "falcon 9 lox vent" for plenty of videos/photos.
The "venting" you usually see isn't LOX, it's condensed water vapor, there are small bleeder valves in the LOX tanks and the fuel line but they are not designed or used for emergency pressure relief.
The cloud is water vapor. But it's not like water vapor is onboard the Falcon and being vented. Boiled-off oxygen is vented, and it's cold enough to cause the water vapor in the surrounding air to condense.
Yes like i said through the bleeder valves in the tanks and the fuel line, but this isn't an emergency release valve, you cannot use it to drain the tanks.
They definitely do vent LOX boiloff.
Yes through the bleeder valves but a bleeder valve isn't an emergency pressure release valve.

Bleeder valves are designed to bleed gas from a system which contains a liquid. They aren't designed to be used to relieve pressure in an emergency situation, infact under high enough pressure they stop working all together.

In a more common setting you see these types of valves on breaks and on heating systems that use hot water, while they do bleed steam they bleed it in order to prevent the accumulation of gas and air bubbles within the system not to control the pressure.

Agreed, I wasn't contending that they are used in an emergency context.
Do you know of any off-the-shelf that are qualified as reliable at the temperatures of cryogenic fluids? (F9 LOX is believed to be at roughly -200 Centigrade, or around 70 Kelvin.)
I can't think of the manufacturer of the top of my head, but there are definitely off-the-shelf options. They are used on the LN2 tanks and dewars you see at a lot of electronics (and I'm sure other) manufacturers, and I'm pretty sure are standard kit for any cryogenic vessel. I think the bigger issue, as one commentor alluded to, is that you might not want to vent whatever is in the rocket straight to atmosphere.
LN2 isn't combustible. LN2 isn't stored at the high pressures that cryogenic LOX and RP-1 are. RP-1 has a flashpoint of 110f, liquid oxygen would turn pretty much every spark into a party including turning a lot of materials which are not normally combustible under atmospheric levels of oxygen highly combustible.

Basically once the cryogenic fuel starts to warm up you have to launch or to empty the tanks, the tanks are pumped out into reservoir tanks which are usually below the launch pad and fueling or emptying the rocket is in general the most dangerous part of the launch/abort sequence. Temperature and pressure variance, vibrations, leaks, and a lot of moving parts if anything goes wrong it can result in a pretty big explosion.

Overall they do have ways to empty the tanks, but this is a controlled pump out of the fuel at pressures and rates that would be as safe as one can get when dealing with supercooled liquid oxygen and kerosene, uncontrolled venting is a big no no.

You can similarly see this on larger tanks of both inert and reactive gases, a small propane tank could have a PRV because a flame out is likely to be less dangerous than an explosion in a BBQ setting, a tanker won't have a PRV because if it vents several tons of propane the result would be as bad or worse than a potential or an actual explosion. LN2 tanks also have PRV's as long as they are small enough to be safely vented but as soon as you go into a large enough volume where venting it is no longer safe whatever fail safe you implement has to be controllable.

Thanks for the info! My experience in the past was mainly with large LN2 dewars, with liquid helium refrigerators added to re-condense the N2 gas back to a liquid, limiting the pressure (and limiting how often you needed and LN2 refill!). We had blow-off valves as a fail safe if pressures got too high if the cooling head ever failed. I'm sure working with cryogenic fuels is a whole different world.

As I commented, it's not that there aren't cryogenic temperature pressure relief valves, just that they may not be what you want!

>As I commented, it's not that there aren't cryogenic temperature pressure relief valves, just that they may not be what you want!

Yeah for sure, if you think about it even if the rocket was filled with inert gas venting it might be very tricky. You have an erect rocket on a launch pad, you have to vent evenly as a pressure relief valve is effectively a "rocket engine" at high enough pressures. If one of them freezes and you don't have even distribution it can topple over.

You also have to vent the tanks in order and at a certain rate I would assume for example you want to vent to top tanks first to prevent the rocket from becoming too top heavy, and you also want to vent it at a certain rate to prevent quick shifts in the balance and weight distribution of the rocket.

liquid oxygen would turn pretty much every spark into a party including turning a lot of materials which are not normally combustible under atmospheric levels of oxygen highly combustible.

Nickel and steel would combust during tests of the SSMEs. (Space Shuttle Main Engines) Even increasing the fraction of gaseous oxygen in the atmosphere can turn all sorts of surprising things combustible. (Like living human flesh!)

It means that if there's a launch failure, you can't just let the stuff sit there. They clearly can pump it out -- the normal sequence for this sort of test (which SpaceX generally does before every launch) involves filling the tanks and then running the engines for only a second or two, and they also obviously need procedures for last-minute launch scrubs, which can happen for a whole bunch of reasons. (One launch was recently delayed because someone steered their boat into the no-go area.)

The main operational consequence to date has been to limit the number of times that they can try to launch within an extended launch window. But if something goes wrong with the fuel-handling equipment, or thermal management inside the tanks, things can get bad.

Interested to know more about the fuels... when I searched for info about Falcon-9 fueling a week or two ago (having briefly conceived of a semi-idle interest in rocket propellants) I read it was RP-1 plus oxidiser - so basically "just" high-spec kerosene. That doesn't jibe with supercooling in my mind... Please enlighten us further.
Liquid O2 aka oxygen is a common and cryogenic oxidizer. Loading cold liquid fuels also has benefits like lower volume, but few things stay liquid at cryogenic temperatures.

PS: When dealing with rockets oxidizer is also a fuel because they need both it and something to burn.

For oxygen to be liquid it needs high pressure and low temperature.
Actually, atmospheric pressure is enough, and oxygen boils at a slightly higher temperature than nitrogen.
(comment deleted)
They chill the fuel and oxidizer to slush, just above their freezing point, to fit more of it into the same volume and save on structure.
How many Falcon 9s have been launched using this fueling method?
At least the last two launches, off the top of my head.
As far as I know that was one of the defining changes of the Falcon 9 Full-Thrust, which flew first in December 2015 with Orbcomm OG2 M2. So, every Falcon 9 this year, plus one.
SpaceX frequently conducts these static fire tests with the payload on board. So unfortunately it's entirely possible that AMOS-6 is lost too. Very sad. Hopefully no one is hurt.
The entire pad may be totaled, too.
Seems likely. Based on the photos we've seen the strongback sure as hell is. It's all twisted and mangled.
Unconfirmed via reddit live stream: payload not on booster, and it was a scheduled test firing so no human beings affected (everyone was well clear).
It looks like it was during fueling, and the payload is lost.
(comment deleted)
From the video it looks like the fire didn't start where you'd typically have your first stage motors.
(comment deleted)
You can see the smoke plume on the radar image from Melbourne airport weather radar

http://imgur.com/a/FQ4qx

Live link:

http://radar.weather.gov/radar.php?rid=mlb

For those of us not familiar with Florida, Cape Canaveral is directly East of Orlando.
Melbourne, FL for those wondering.
Melbourne, FL, United States :)
We sure that's a smoke plume? There's a lot of radar activity over Florida presently due to the "OMG first hurricane in 11 years!!!"[0]

[0] I live in and am in the Big Bend area, so the news, tweets, emails, and texts about the storm are a bit ... much.

Just read no injuries. That's awesome. I mean, bad for the rocket, but hooray for no human loss of life.
I am surprised that no one has mentioned whether or not this failure is related to SpaceX's first time reusing a rocket. If so, hopefully this isn't too much of a setback!
This is a brand new rocket. The reused rocket will be used for the SES-10 mission.
The reused rocket will be used to launch SES-10 later this year, not AMOS-6.

Source: http://spacenews.com/spacex-to-launch-ses-10-satellite-on-re...

Question is whether they will have a launch pad at that time and how long investigations will last and until a fix is deployed. May well be that there are no more launches this year.
Yeah the last time they lost something it was a 6 month freeze on launches.
(comment deleted)
"This rocket was scheduled to launch the Amos-6 communication satellite, which among other functions included the capabilities for Facebook to spot-beam broadband for Facebook’s Internet.org initiative"

Bad karma

No payload on rocket while hotfire test.
(comment deleted)
Payload is sometimes mated, just depends. No word yet on this payload.

https://twitter.com/NASASpaceflight/status/77134295079838924...

UPDATE: payload lost. https://twitter.com/SciGuySpace/status/771352111657385984

That confirmation is so strange. Why would you confirm something with the screenshot of notepad "officially" when you have a twitter account?
He's a space news reporter at Ars Technica. It makes sense they'd be one of the first to get the SpaceX news release. It has too many characters to fit in a tweet, and even if it could fit, embedding an image of the text sets it apart from the tweeter's introduction and makes it more readable. It's standard practice on twitter to use images for blocks of text.
> It's standard practice on twitter to use images for blocks of text.

Yep. The best of 1997, Web 2.0 style.

To keep it out of search results. They can delete the image of the text whenever they'd like, and poof
> Why would you confirm something with the screenshot of notepad "officially" when you have a twitter account?

I figured it was because they didn't want to be constrained by a character limit. It allows them to use very specific language.

"SpaceX can confirm that in preparation for today's static fire, there was an anomaly on the pad resulting in the loss of the vehicle and its payload. Per standard procedure, the pad was clear and there were no injuries."[0]

[0] https://www.reddit.com/live/xix3m9uqd06g

a facebook satelite doesn't sound too good anyways.
Well, right now it is more of a faceplant satellite.
It's hard to imagine that this sort of scheme is the best way to provide broadband to most parts of Africa. Why can't they just put microwave links on their cell towers? This is some sort of speculative play by FB to bypass local influences.
They got rejected in India by activism among users and the regulator TRAI. So, I also wondered the exact same thing.
According to sources (twitter), they where F-3 minutes so no-one should be out on pad, payload was not loaded either.
According to sources (twitter), they where F-3 minutes so no-one should be out on pad, payload was not loaded either.
(comment deleted)
The smoke looks fairly white and uniform. That suggests it's coming from the rocket fuels, not a burning building or other facility. To me, that means the safety protocols held, limiting the spread/damage. Hopefully nobody was hurt.
For all developers of anything. Repeat this mantra.

This is why we test, this is why we test, this is why we test.

It is especially important to repeat this mantra around management types who want last minute builds before going in front of important customers, because the light blue button looks SO much better then the dark blue button...

I mean, it's not like SpaceX doesn't test. They missed something, and unfortunately this time it caused a total loss that they're going to have to answer to. But yes, you're right; we recently implemented a two-week demo lockout policy at my office. No changes are allowed being committed closer than two weeks from a demo date, and two or three guys are dedicated to just testing and shaking down the system. They're the only ones allowed to call for a bug fix, and they're only allowed to call for a bug fix. It seems as though management has finally realized that missing one requirement during a demo is much, much better than having the whole enchilada crash spectacularly.
The problem is that software works the same every time, hardware doesn't have to (for example, if there was some kind of wear in the umbilical, for example).
The Antares explosion footage should accompany that mantra.
This was a test which had followed previous testing.
My mind goes to the debugging stage after the dust has settled... how would a rocket scientist determine the cause of a RUD like this?
About the same as you debug an issue from having just a tracelog. The rocket and pad equipment have quite a few sensors that record information, so you have quite a lot of data afterwards to spot anomalies (you could also compare to previous successful launches / static fires to spot them). Admittedly, that doesn't really explain how they figure out some stuff, e.g. with CRS-7 the problem was with a strut holding a high-pressure helium vessel in the second stage. They likely would have seen the overpressurization of the tank and there are not many sources for that pressure, but pinpointing it to a piece that doesn't even exist anymore ... beats me.
I don't recall the source, but they identified the CRS-7 strut by using several mechanical vibration sensors placed on various parts of the rocket.

The deflection / explosion reaches the sensors at different times, and with precision timing, it enables you to reverse pinpoint where it must have originated: the failing strut.

they also have pressure sensors in the tanks so the timing of pressure differences allowed them to determine that the tank came loose before it burst.
They probably brainstorm through all the possible scenarios that could have caused a particular condition and then see if the data matches and they can prove it as best they can.
Not a rocket scientist; just an electrical engineer with plenty of failure analysis experience. Here is generally how I would translate my process to this case.

As a sibling mentioned, there will be lots of telemetry about what was going on with various systems during the incident. That data may indicate a few specific points to start, but it will mostly be useful for putting other facts into context. In parallel would be an effort to comb over the pad, recover as much as possible of what remains of the rocket, and try to identify what parts are what. There may be some obvious forensic evidence of parts to concentrate on, but in all likelihood the analysis will be guided by a combination of analyzing the telemetry and the damage patterns on the remaining pieces. Hopefully the combination will narrow the probable causes down enough to concentrate analysis on a couple-three theories.

Once plausible and probable theories are identified, teams will start to drill down into what event chains could cause them and look for evidence to confirm or reject those. Some of that will involve destructive analysis of the wreckage; you can glean a surprising amount of information from high-magnification images, x-rays, and sections of what otherwise looks like a twisted hunk of scrap metal. Some will involve modeling; no doubt the engineers who designed the engines have some sort of model they used to test it, and the parameters on that model can be varied to create various out-of-spec conditions. Some will be design analysis, to see if some previously-unseen corner case could have caused an otherwise-conformant system to fail catastrophically. Test data on the involved systems will be analyzed to look for any anomalies that might have been passed off at the time but that might be significant on hindsight.

Assuming possibilities still exist and are not sufficiently firm, or even just to be extra thorough, a few possibilities may be tested in practical (and possibly destructive) tests. In other words, try to blow up (or simulate blowing up) another engine in a controlled manner. That may or may not be possible, depending on what the root cause is; a parts tolerance issue, for example, would be practically impossible to recreate unless it were possible to manually change affected parts to match the suspected tolerances that caused the failure.

I'm sure an actual rocket scientist will come along and provide more detail. My work was limited to exploding caps and FETs.

If you're interested in this, you would enjoy reading the analysis of the breakup of the shuttle Columbia (not the earlier Challenger analysis, which is in itself interesting). Give yourself an hour.

The basic link is: http://www.nasa.gov/columbia/home/CAIB_Vol1.html

Start at chapter 2, on page 27. Highlights: "lock the doors" (page 44), the hunt for debris (page 45), reconstruction (page 47, 72, 74ff). It's really an amazing engineering detective story.

The fault tree analysis methodology is on page 85ff.

Looks like they had a payload go with it. Sad.

> "SpaceX confirm Amos-6 was aboard the Falcon 9 and was lost in the explosion."

https://twitter.com/NASASpaceflight/status/77135388623807283...

There were no personal injuries, so I can't really be sad about a piece of electronics.
That's tons of work. Lots of heart and soul went into that. I can still be sad for people even if we didn't lose people.
Not to mention money. $200MM investment into space, gone.

https://www.reddit.com/live/xix3m9uqd06g/updates/613f262a-70...

The money didn't disappear, it paid the people working on the project all the way down to the miners who dug up the minerals and the fast-food workers who made the cheeseburger that engineer bought that one time.

It did fail to earn a return, however.

You're advocating the Broken Window Fallacy.

http://www.investopedia.com/ask/answers/08/broken-window-fal...

No, in this field sometimes stuff blows up unexpectedly and it's part of the cost of doing business. It's like feeling bad for a life insurance company when someone dies.
(comment deleted)
Not really, since he did note that the money will fail to earn a return.
Not quite. The Broken Window Fallacy hinges on the fact that the son breaks the window for a reason that doesn't further any other economic cause. That is, he didn't break the window for research, so nothing new was gained from it. Resources were merely shifted from the shopkeeper to the glazier.

When it comes to innovation, however, failure is often the impetus to more efficient design. Not only was that $200m shifted from Musk/Investors to SpaceX, et al, but it also went to informing the process and improvements for the entire project, and future projects by other companies.

We wish we could learn these things more cheaply, sure. But that doesn't mean the world or company would have been better off in the long run had the incident not occurred -- it's too soon to tell, and we might never know.

You can't say that all mistakes are worthwhile just because they reach us that we shouldn't make those mistakes... We already knew that. That wasn't research, it was a prelaunch test, and it failed.
That's not what we're learning. We're learning how not to make those mistakes. That yields: 1) Lower risk launches in the future through better process and better design 2) Possibly cheaper/better hardware redesigned due to the issues uncovered today.

Yes, it is probably lower ROI than that payload getting into space, but it does mean it's not a total loss and not simply a broken window fallacy.

It's more akin to if the window maker also tried to learn/test harder to break windows with each one she installed. Then, each broken window would be an experimental outcome instead of just a lost window.

There's no way to get around making mistakes no matter how careful you are. It's inevitable for non omniscient beings.
Even if they are (which is apparently disputed), pointing out that someone is using a fallacy does not make them automatically wrong. Next time, you should try saying something more substantive than "Fallacy!"
> using a fallacy does not make them automatically wrong

I thought that is precisely what using a fallacy makes you. It's what the word fallacy means. It essentially means a "false statement".

Fallacy means faulty reasoning, not a false statement. It is possible to use faulty reasoning to arrive at a correct statement. For example, "The sun rises each day, because if it didn't then everyone would die." That's the fallacy of an appeal to consequences, yet the conclusion (the sun rises each day) is correct.
Well, everyone would die, though :)
Of course! The only incorrect thing in there is "because."
This line of reasoning is itself a fallacy. http://www.fallacyfiles.org/fallfall.html
Furthermore there are other forms of reasoning than pure deduction. If you allow probabilistic inference, which is how humans generally operate intuitively, many deductive fallacies turn into probabilistic theorems.
I'm kind of skeptical about this alleged fallacy, where I have to admit that I'm not an economist. The usual formulations of it seem to be something crucial missing.

I've heard that many producers of goods deliberately introduce failure points, e.g. in electronics by using cheap solder or capacitors with a limited lifetime. Buttons also fail way too easily. Or think of batteries that cannot be replaced. Do they all commit this fallacy, too? Do they harm the economy and therefore indirectly also themselves?

Or is it a matter of how long the window is used before it is broken? If so, how long? You could also make nearly unbreakable glass (buttons, rockets, etc.) but at very high costs for the company and therefore also the consumer. What role do the costs play in all of this? Is it an equilibrium? When does the fallacy start and normal 'crap product' cycle end?

I've never seen any explanation of this alleged fallacy that answers any of these questions.

(comment deleted)
My reading of it: there's a continuum in production philosophy from throw-away culture to built-to-last products, and the broken-window fallacy (which Krugman, IIRC, doesn't think is a fallacy at all) is the extreme end of throw-away culture.

Building to last means high up-front costs, little flexibility (think of the Empire State Building and how much it must have cost to install air conditioning in it), but beautiful products with low total cost of ownership; throw-away goods are low on up-front costs, and they make it easy to respond to new technology, but they have a high total cost of ownership and they tend to be pretty ugly as well. A society which focuses on quality will be wealthier and more beautiful (look at Europe's low GDP and high standard of living), but one that's constantly rebuilding junk will be more equal; every 1970s Volvo still owned by an old-money family in 2016 is a Ford assembly-line worker without a job, or thereabouts...

> I've heard that many producers of goods deliberately introduce failure points

There are a lot of apocryphal claims like this, but little evidence. Most manufacturers design for an expected life of the product, and making it last longer than that is a waste of money and resources.

For example, you could design a computer to last for 20 years, but what would be the point? Computers go hopelessly obsolete in about 5 years. The only people who care about longevity of it are a handful of collectors. Fashionable clothing is not made to last because people don't wear out-of-fashion clothes. It's pointless to make them to last. Cars are designed to last for 10 years. Airliners are designed to last for 65,000 flight hours.

Products that are useful long term are usually made to last, like tools.

> Products that are useful long term are usually made to last, like tools.

Except they rarely are nowadays either. Lightbulbs would be a common example, but so would be cheap construction tools, kitchen tools, knives, etc. all designed to last for few uses and then break, so that people buy a replacement. The argument of "waste of money and resources" only holds for a single company, but not for the economy as a whole - it doesn't factor in the costs (and energy waste) of replacement and of dealing with the garbage, nor does it factor in the ecological damage created by unnecessary manufacturing.

It all boils down to the standard short-term, greedy optimization (in algorithmic sense) of the market economy, giving you perfectly legitimately sounding reasons to keep being stuck in a crappy local minimum.

Any wrench or hammer or saw you happen to buy will easily last you a lifetime, even if you use it constantly. I've seen some really cheap silverware (stamped out of sheet metal, and practically unusable) and some shoddy knives, but spend even a few dollars more and you can easily find something that will last for many years. The inexpensive silverware I picked up at Target a few years ago is solid cast stainless steel, and I have no doubts that it will last a lifetime; I doubt I could break a tine without tools. A cheap cast-iron skillet can easily last multiple centuries.

Most furniture is actually pretty shoddy, made from MDF and glue, but even so I've managed to find some that I'm happy with. My first office chair was not very good, and failed catastrophically in a few years, which was annoying. I managed to replace it with a much nicer one that I trust to last for decades. (It was rather overpriced though; I won't break even for something like 50 years. On the other hand, I can sit in it all day without hurting my back, which is more important to me.)

I have 8 year old computers that are perfectly good (aside from a hard drive that had to be replaced a year ago, and a new power supply the year before that), a 5 year old laptop that does all I could ask, etc. Computers are no longer advancing at such a breakneck pace that they're obsolete in a year, and the operating systems no longer have a lifetime measured in years either. I expect them all to last quite a bit longer.

I'm sure my washing machine will fail in a decade or two, probably the motor will burn out or a capacitor in the digital timer will fail. Neither are impossible to fix; the only question will be whether a newer washing machine would have enough extra efficiency to make replacement a better choice than repair.

Cars routinely last 200k miles or more, and replacement parts for most cars are easily had.

I would say that most products have a pretty good lifespan. Obviously my experience isn't universal, but I would say that most things are built well enough. It is worth paying attention to what you're buying, but most products are not actually intended to be disposable, except in areas of rapid technological change.

On the other hand, you could look at something like the jet turbine or transmission in an attack helicopter or tank. Those have a very definite lifetime, and a very strict maintenance schedule, and they're measured in hours, not years. If you do all the maintenance correctly, your jet turbine might last 500 hours (or some similar number, I'm not very familiar with the specifics) of use. Once you've used it that many carefully-logged hours you take it out and replace it with a new one. Maybe that's what you're thinking of? It's certainly expensive, but in military hardware you want to extract the maximum possible combat performance from everything; lifetime is pretty far down the priority list.

Broken window fallacy is talking about the total (net) economy. It's perfectly possible for one person to come out ahead while the economy as a whole goes nowhere or even down.

For instance, if you are the only window repair person in town and you go around breaking other people's windows, you will certainly profit. But the amount you profit will be completely offset by the window owners' loss. (Presuming you do not go to jail, of course.)

No, I said that the money was spent, not lost; it didn't just vanish. They didn't bury it in the back yard and then forget about it, or set it on fire. They've definitely lost future income though, even if they rebuild the satellite and launch it successfully a few years from now. (I know that you usually build a test version of your space hardware, for tests that might be destructive. I wonder if any of those have ever been launched insted?)

It was also spent on something they wanted to spend it on, rather than something they were forced to spend it on. It's still an interesting point though, because they spent it on something that carried significant risk. Does spending money on a glass window, which carries the risk of a broken window, work in the same way? I've not considered the broken-window fallacy from that perspective before. Perhaps it's not, for the same for the same reason that breaking a window is considered a crime and an exploding rocket generally is not.

Is that the manufacturing cost, or does it include the development cost?

Making another should be cheaper than making the first one; and this is Amos-6 -- how different is it from Amos-5?

I would guess they lost a lot less than the $200m initial investment, which included design & development of both the satelite and ground control systems. My assumption would be that only the satellite itself was destroyed in this explosion.

From the quote below, they reference $85 to cover launch, insurance, and 1 year of operating costs. Presumably that insurance would cover some of the costs of an event like this?

Quote from an article on this (http://www.globes.co.il/en/article-1000759794)

Spacecom Satellite Communications' (TASE:SCC) board of director has authorized management to sign a contract with Israel Aerospace Industries Ltd. (IAI) (TASE: ARSP.B1) to build and buy the Amos 6 communications satellite for $200 million. IAI will build the satellite and its ground control systems, and will provide operating services. The company plans to launch the Amo6 in the first quarter of 2015, and its operating life will be at least 16 years.

Spacecom estimates that the cost of launching, insuring and one year's operation of the Amos 6 will be $85 million. The company has to pick a launch company. Spacecom said that it will seek financing for the Amos 6 from IAI and foreign sources.

I would be quite surprised if the payload was not insured. Doesn't make it any easier on the people who poured their lives into the effort, but the owner will not be out any substantial investment capitol; just potential earnings until its replacement can be pushed up.
The actual chips may have to be made again, but code that has been written once should be fine to just load on again.

Yeah it sucks, but it could have been a lot worse.

One could also feel sad for the (possibly millions) of people in Africa who were depending on this satellite launch to give them internet service. The difference between having internet, and not having it, is huge, especially in the developing world. This has real human consequences.
Tell that to the people in Africa who will soon have no internet access once AMOS-2 fails.

This payload, AMOS-6, was a replacement for the soon to fail AMOS-2 which among other things is part of a Facebook initiative to provide internet access in parts of Africa.

I am not aware of any replacement in the works for it, so that service may just shut down, and who knows if it will ever start back up.

So while this might be "nothing" to you, it has actual repercussions for people.

Not to mention the effects on the Israeli company that made the payload. Insurance may or may not pay for the satellite itself (I don't know), but even if they do, they were also relying on income generated from running it.

Actual humans working for that company may now be impacted.

It's a real bummer, also don't forget that now that SpaceX has had two payload losses in two years will mean that their insurance is very likely to go up.
> provide internet access in parts of Africa

If you mean internet.org wheter it's a loss or not is debatable

Internet.org was only a part of AMOS-6's planned internet bandwidth. Per Wikipedia "36 regional spotbeams with a throughput of about 18 Gbit/s — on Amos-6 to provide service for Facebook’s Internet.org and a new Eutelsat subsidiary focusing on African businesses. Costs would be divided in approximately equal shares between Eutelsat and Facebook."
Agree. "Sad" is the PC thing to say when something like that happens. In most people's lives it's a big "who cares". [1]

The fact that you were downvoted (at least when I saw your comment) indicates people don't agree with what you are saying and is an example of trying to enforce a particular type of thinking on someone else as far as what they should think or feel.

[1] Of course I wouldn't say "who cares" directly to someone involved in the project in some way then you extend your condolences.

Why the need to comment on perceived downvoting injustice? Is it that big a deal? Is there a downvoted support group somewhere? :P

Anyway, here's the scanner feed of Kennedy Space Center Communicationshttp://www.broadcastify.com/listen/feed/705

He was downvoted for only looking at first-order effects: People harmed during the fire.

And ignoring second-order effects: People harmed because of the results of the fire.

The second group of people is not less important than the first.

Why is it necessary to feel sad and to even consider all of that? That is my point. What bothers you bothers you and what doesn't is fine, less to think about. You can't go around being bothered by everything 'bad' that happens in the world to someone else. I don't think that is healthy in any way. By "bothered" I don't mean indicating in words that something is sad. I mean actual concern as if it impacts you or someone you know personally.

Sorry but I can't relate to this type of tragedy at all. I can relate to someone getting hacked and losing their business or customer information though. That pain I can imagine. And to hear someone say to me "that's sad" about the latter when knowing they don't have any idea what it is really like to me is just lip service and almost patronizing.

If you don't feel sad about something, then don't.

But you don't have to go around saying "LaLaLa I don't feel sad." Just don't say anything.

> I can't relate to this type of tragedy at all

This was a major part of a charitable initiative to bring free internet access to Africa.. That's a lot of lost opportunity for business, education, health care..

From a technical perspective, the laser-based communications network being developed by FB for Africa is super cool. I'm sad to see this delayed. Even though it doesn't affect me directly, it hits my inner nerd right in the feels.

Insurance. People don't spend tens of millions of $ sticking electronics on top of giant fireworks tubes without hedging the risks involved. If this is getting you int he feelz then you need to get out a bit more, this is merely a hiccup.

Sure, it will delay some things in Africa, but we can't easily say whether that delay will be good or bad. For all we know people will be better off due to the delay as it may be less disruptive of existing economic structures, notwithstanding their long-term inefficiency. You're assuming incorrectly that the consequences of this satellite deployment would have been all upside and no downside, but such effects are not reliably quantifiable.

Hey, why are you letting this bother you so much? Don't let what bothers other people bother you, I don't think it's healthy to want so much control over other people's emotions, and surely you'd have less to think about it you stopped letting it bother you so much?
You wouldn't say it directly to someone involved in the project, but you'll say it in a public forum where there's a decent chance some of those people will see it?
Exactly. That is the problem with the entire PC thing. You have to walk around on eggshells being worried that you will offend someone and never say what you think. I mean seriously. Be worried that someone who worked on a rocket project for SpaceX might be offended by what I say? I am supposed to not say anything because of that?
There's nothing wrong with softening your language a bit just in case someone who happens to be involved sees what you write. If not being a jerk means you can't speak what's on your mind, then that says more about your mind than it does the world.
No, you're supposed to understand the concept of "manners", like a functional adult, instead carrying on like an overgrown toddler.
Of course I would say it to someone directly involved in the project. I've had projects I worked on go south through unavoidable externalities, of course I felt disappointed but there' no irreversible loss as when someone dies or is permanently disabled. Save your emotions for things that actually matter, an economic/operational setback doesn't qualify.
On any project like that many engineers involved will have invested more than regular hours and pushed their work life balance - losing it all a couple of days from launch into space - that's harsh dude
Can you prove any of this?

Knowing how smart and methodological Musk is, I wouldn't be surprised if the rule at the office is actually to leave your desk and go home at 5pm sharp.

You are prone to make more mistakes when you tired and overworked and I cannot imagine more important place not to make mistakes, than building a rocket.

Wouldn't be surprised if some overworked engineer made a mistake.
Wouldn't surprise me in the least. I know Musk claims that "working at Tesla/SpaceX is like being in the special forces", but there's no way that every employee at these companies has the work capacity to stay 100% focused through 80 hour work weeks all the time.
I agree. I bet that the 80+ hour work weeks are going to result in death someday.
Thank you.

So they are paid overtime for this, is that correct?

Not if they're salaried, which most of them probably are.
:O.

So US doesn't have paid overtime on a regular salary work?

Not if you're covered by the "white collar exemption", which any engineer-type is going to fall under. Folks working 80 hour weeks at startups don't get any overtime for it.
They why are they being allowed to work for free?

Is that even legal in USA? Aren't they obligated by OSHA and USDOL to be paid for the time they work??

Not if you're covered by the "white collar exemption", which any engineer-type is going to fall under. Folks working 80 hour weeks at startups don't get any overtime for it.
Engineers are not paid overtime. Technicians often are paid overtime.
Not only that, it's also not really a secret that SpaceX attracts people who want to enter the space industry. You get experience there, you move on.

Yes, they have experienced people too, but overall, turnover is high for a reason: https://hackerlife.co/blog/tech-employees-turnover/Los-Angel...

> Not only that, it's also not really a secret that SpaceX attracts people who want to enter the space industry. You get experience there, you move on.

Which is exactly the kind of people that SpaceX does not want and who should not apply there. The turnover is high for a reason indeed.

If SpaceX wants experienced people, they should try the whole stick of decent hours and decent pay.
Then again, you also have the payload manufacturer and other involved parties, which are not managed by Musk. Either way, a lot of hard work has gone up in flames, and people will be understandably disappointed.
It's the life that they've chosen. It comes with ups and downs.
Most of the stuff on which we engineers work ends up in the bin anyway. This is just more spectacular than some manager showing up one day to announce that this XXX M$ project, the one you and many others spent your last 2 years regularly overtiming, is cancelled because mblllmbllbllsometopmanagementfuckedupmbbllmbllblll. I just mention this case because it was the most ridiculous that happened to me but a large majority of others projects I've worked ended up in the bin a way or another, or should have because they are designed from scratch to be useless.
I think it's unfair that you are getting downvotes for this statement.

All you said was that it is good to have some perspective. Yes, it sucks to lose the rocket and the satellite. But to put it into perspective, the husband of the lady sitting next to me as I type this (and a friend of mine) is currently sitting in the shelter at the launchpad. He texted her to let her know he was alive. He can't give out any more details.

Have some perspective people.

I agree. You could argue it could always be worse (2 deaths vs 1, a death of a loved one vs. a stranger) but I'm sure the level of badness has a steep curve before human loss is considered.
How close is the shelter to the tower? That's gotta be terrifying. I'd have thought people would need to be pretty far away just to avoid permanent hearing damage in an event like this.
I could have formulated that better.
I know nothing about space - why are they doing tests, where the rocket could possibly explode or fail, with the payload on board?
The payload is probably loaded in a clean-room type environment.
That was my first reaction. I thought they could create a dummy payload with the same dimensions and weight. But, then... unless the weight distribution of the payload is important.
Loading the payload isn't a quick thing, it takes time and a lot of careful work to do. So doing it before the tests can save a lot of time and I'm guessing money.
Loading the payload also requires a load of work moving the vehicle about etc. If you want to be certain you didn't damage the vehicle in any way during all of that then you need to test again once the payload is attached.

Of course normally a failure during a test would be something like minor thrust fluctuations - not a complete loss of vehicle and payload!

Understood that mating the payload is a lot of work, but it is one hell of a lot less work than building a complete new payload.
I think this kind of test is generally just looking at the performance of the engines. Certainly highly unusual for the rocket to explode as part of this kind of test.
Same reason anyone else does tests: Find potential and actual issues earlier than in production. The static fire they were doing is sort of a short integration / smoke (pun not intended) test whether the rocket as a whole (whether the payload is attached already depends on the customer; it reduces time between test and launch, though) works properly. Other tests are part of sort of a pre-flight checklist directly before launch and there have been launch scrubs due to issues at that point already. Every single engine is also fired for the full duration of the flight prior to installation in the rocket.

There's a lot of testing, both in isolation and in integration with other components. The risk of the rocket exploding should actually reduce with each test. Note also that in this case the malfunction was most likely with the pad equipment, not the rocket, so not doing static fires would in this case probably just have meant that you'd have an explosion at launch time one day.

EDIT: The static fire is more of a test for launch procedures, apparently:

“The goal of the static fire is to provide a dress rehearsal for the launch team, culminating in a three second firing of all nine of the first stage Merlin 1D engines to validate the health of the rocket.” (https://www.nasaspaceflight.com/2016/09/falcon-9-explodes-am...)

Makes sense in that the rocket itself is tested quite a bit beforehand already.

Is launch pad designed to take a hit like this? Or do you think it would have caused a lot of damage to Cape Canaveral.
Unlikely, it wouldn't be much of a launch pad if it couldn't handle failure modes for rockets (aka bombs with a hole in one end). Compare to this Saturn V launch.

https://www.youtube.com/watch?v=DKtVpvzUF1Y

The video points out two of the protection features, but they wouldn't do so well with a RUD.

The tower features - the hold-down arms, etc - are painted with a sacrificial paint. The idea is that it's the paint that chars and burns, rather than the tower features.

Then there's the water deluge system.

In the video, the rocket and exhaust is clear of the tower, and the fires are out, within 30 seconds. Neither the sacrificial paint nor the water deluge are designed to handle long-duration fires from a RUD.

After a 2014 Antares rocket failure, the launchpad at Wallops Flight Facility took 1 year and $15 million to repair [1].

[1] http://www.space.com/31412-virginia-launchpad-private-rocket...

It would be incredibly difficult to armor a launch pad to survive an explosion on-pad, so they aren't. The real solution is to have multiple launch pads (the space shuttle had three), so that if one blows up on the pad you have backups you can use until the blown up one is rebuilt.

The energy released in the first few seconds of a controlled launch is not remotely comparable to the energy released by an entire rocket blowing up simultaneously. Also, with an explosion, the entire rocket, along with parts of the strongback and other structures it's attached to, become shrapnel. Superheated water exhaust is a lot easier to protect against.

Launch pads usually are not built for rockets exploding on them, but as the pad itself is mostly a concrete plane it should be mostly fine. Controlled burn of the propellant, a.k.a., a launch, is obviously fine as well ;-)

Facilities and other things on the pad may not like it, though.

They already said that their Horizontal Integration Facility (where the rocket is put together prior to erection) is intact, as are the tanks on site. But the strongback looks mangled and may well be destroyed. It's also right beside the rocket, so the most likely casualty in such an explosion. Other pads and facilities are most likely far enough away that the only concern is debris from the explosion landing there.

A rocket contains the energy of a mini-nuke. You could not design a launch pad to withstand an anomaly like this.
The pictures of the aftermath speak for themselves. The strongback is a mangled wreck, and all of the fuel tanks blew up. The vast majority of the launch complex, expressed in economic terms, is going to need to be rebuilt.
They do the tests to validate that this rocket won't turn into a cruise missile, headed for the nearest residential area. Not an impossible scenario. Lots of energy stored up in one of those rockets.

Just like when we test software, they want to have everything as close to what they would expect on launch day as possible (2 days from now). All the stresses, the structure, etc. That means putting stage 2 and the payload on top of the main rocket before the test starts. They probably did a lot of tests before the payload was on board as well, and those didn't reveal whatever flaw has caused this issue.

Yes, it sucks that the payload was lost but there will have been insurance to cover the loss. No human lives lost, no cruise missile scenario, no out of control fire... this is the best case scenario for a rocket failure.

> They do the tests to validate that this rocket won't turn into a cruise missile, headed for the nearest residential area.

In that case the rocket has a flight-termination system, though, which should activate as soon as it veers too far outside the planned/expected parameters of the flight.

This discussion reminds me of the failure of the first Ariane 5 launch. The flight-termination system was activate erroneously, and the payload was not insured. Ouch.

http://www.esa.int/esapub/bulletin/bullet87/cavall87.htm

https://around.com/ariane.html

https://en.wikipedia.org/wiki/Cluster_%28spacecraft%29

That's not what happened - the rocket went out of control due to a software bug allowed by over-reliance on unit testing, and was quite properly deliberately destroyed.

But yes, Cluster was run on the cheap, hence the use of the Ariane 5 test flight, and didn't have insurance.

"Over-reliance on unit testing"? I have a somewhat different read from the inquiry board report, which says it was a "systematic software design error." Yes, inadequate testing, and a belief the working code for Ariane 4 meant it was validated for Ariane 5 were certainly contributory, but I think it's unfair to single out an over-reliance on unit testing.

The board argues that there was a bias towards believing the software does not have an error. Thus, any out-of-range value is interpreted as a hardware error, which means the CPU should shut down.

There was a decision to not include Ariane 5 trajectory data in the SRI requirements and specification. Thus, while tests were rigorous at the equipment ("unit") level, and there were system tests, they didn't test that case. This is test design failure.

In addition, the board says "the review process was a contributory factor in the failure."

I can see how those can be aspects of "over-reliance on unit testing", but it doesn't explain, for example, how some of the variables from Ariane 4 were protected from overflow exceptions but others were not.

You have quite fairly picked me up on a cheeky exaggeration.

Lots of things had to go wrong to cause the Ariane 5 failure - including bad handling of overflow, as you mention. But to my mind, the universal last line of defence against any kind of mistake is an integration test: put all of the parts of the system together, feed them real input, and verify that you get correct output. Arianespace did not do that.

Well, until they actually launched it. It was a test flight, right? It proved to be an essential and very effective test.

That means there was an under-reliance on system testing, which I agree with. It doesn't imply there was an over-reliance on unit testing any more than it implies there was an over-reliance on analysis or over-reliance on the expectation of developing bug-free software.
If anyone's at all interested in risk analysis or robust software, it's totally worth finding a decent writeup on the Ariane 5 failure --- it's a fascinating postmortem in just how many unexpected little things together added up to a big boom.

Everything is incredibly obvious in hindsight, of course, but making things obvious is largely what hindsight is for.

And once you've finished reading that, go look up the Therac-25...

Oops, yeah I recalled that the subsystem that caused the problem didn't need to run after launch, so I was thinking it fed incorrect data to the auto-destruct system, not that it actually went off course.
For more context, all (edit: American!) rockets for decades have had onboard self destruct systems and a "Range Safety Officer" on the ground whose entire job is to determine if and when to deploy this self destruct system.

The cruise missile scenario is highly unlikely as the rocket itself would be destroyed soon after leaving its intended trajectory.

Not all rockets:

https://en.wikipedia.org/wiki/Intelsat_708

(China, CZ-3B, Intelsat 708 payload, the launcher flew off-course and crashed on a village in 1996: by some estimates 200-500 civilians were killed.)

All rockets launched in the US have to have one, though :-)
That links to http://www.thespacereview.com/article/2323/1 where one of the comments is:

> Like Russian vehicles, there is no flight termination system that receives ground commands onboard Chinese launch vehicles. Only US and ESA launch sites have such a system. Correction, Falcon 1 did not have such a system for launching on Kwaj.

That's because usually the US and ESA/France launch from densely populated areas.

The launch pad at the Kennedy Space Center/CC is effectively about 50 mile from downtown Orlando, Baikonur is in the middle of nowhere.

Okay, rockets that can go very high, can still crash very far away.

An unpredictable, malfunctioning rocket could still million-to-one itself onto a school bus filled with children, halfway across a continent.

Phlarp's original comment was about 'a "Range Safety Officer" on the ground whose entire job is to determine if and when to deploy this self destruct system.'

The text I quoted implies there is an onboard flight termination system, even if there is no Range Safety Officer who can send external commands.

FWIW, a part from an exploded rocket, like the engine, could still destroy a school bus filled with children. The odds are very hard to estimate, and made more complicated in that there are few failure modes where a rocket failure halfway across a continent, at supersonic speeds, would reach the ground without breaking long before.

Congratulations on replying to me. I'm sure you're feeling good about it. I just wanted to let your know that I noticed, and feel special too.
How often is the self-destruct mechanism tested?
Very often. Air Force range safety guidelines require a million dollar flight termination system and most of that cost is testing and quality control. I don't have an authoritative list of all flight terminations but it happens at least once every few years whenever a rocket fails to follow the set flight path.
I also have to suspect things like isolated, dedicated and redundant comm links with the flight termination system are part of these guidelines. Perhaps even a "dead man's switch" that terminates the flight if this separate communication system loses signal at some point. (Even if this means a costly false positive or two~)
For the Space Shuttle, the flight termination system had two huge 10KW transmitters to send the signal. That would get through despite damaged receiving antennas, noise, or jamming.
10kW? That sounds like a lot of energy to dump into air. Certainly unhealthy for operators or people near the tx site.
Less than the typical TV or FM broadcast tower, and remember there's a lot of empty space around the launch sites. Even ham radio operators in the US can run at 1.5kW peak.
You can get up to 30 kW (pulsed) on a MRI all directed at your body.
As an example, the strut failure in the falcon 9 a while back didn't explode from the failure, that was the flight termination system that actually caused the fireball when things started going south.
Imagine how stressful that job would be when astronauts are the payload!
There were no self-destruct systems on the shuttle orbiter, though there were on the solid rocket boosters and external fuel tank. They were used only once, on the Space Shuttle Challenger after the orbiter broke up, but the SRB's were still burning in an uncontrolled manner.
Yeah SRB's don't really have an off switch unfortunately.
Don't worry, unlike the space shuttle, rockets have escape mechanisms. In that specific case, the astronauts would probably have survived as the emergency rocket thingies would have fired and evacuated them far from the explosion before it could reach the capsule.
How are these triggered? Motion-sensors?
(comment deleted)
Nit: flight termination system does not cause the rocket to magically vanish. It will still continue on its current trajectory and impact Earth somewhere. It just causes the rocket to stop thrusting and, IIRC, disperses the fuel/oxidizer so that we don't end up with large quantities of fuel/oxidizer in a small area on ground.
Indeed, all it does is break up the rocket. Depending on when it happens during the launch we might still end up with a fireball near the ground (unlikely to be near anything inhabited, though, as there's a lot of free space around launch sites) or with a quickly disintegrating rocket because it's already at high speeds.

There will still be debris, of course, but I guess the reasoning is that it's preferable to have relatively small debris, than one large piece of exploding debris.

Size of debris is not that important: lots of small pieces will cause a similar amount of damage as the same mass in a large piece.

Two important roles of flight termination are: 1. Cause the rocket to stop thrusting (and thus prevent it from thrusting out of range safety exclusion zone). 2. Cause the propellant tanks to be destroyed. This prevents the propellants from causing a large explosion on the ground (when the tanks hit the ground) in preference to a conflagration in the air.

(comment deleted)
That doesn't mitigate the danger really.
It too should work. Just like rockets shouldn't explode while loading fuel before a test firing.

But of all systems I never want to have to test in production, the flight-termination system is at the top of my list.

Nitpick: you're thinking of a ballistic missile. Cruise missiles are more like exploding fixed-wing drones. The V-1 flying bomb was a cruise missile, and the V-2 rocket was a ballistic missile.
If a fully loaded (cryogenic fuel + oxidizer) SpaceX rocket did a belly flop into downtown Orlando I suspect this distinction would be quite meaningless to the locals!
Sure but wouldn't most things be meaningless to them at that point? S/he said it was a nitpick, i appreciate the info.
'Cruise missile' has always sounded cooler than ballistic missile, imho. But you are 100% correct according to Wikipedia.
Not sure what exactly are you nitpicking here - ballistic missiles have propulsion for a short time after lunch and afterwards continuing on a ballistic (hence the name) trajectory towards the targets. Cruise missiles have propulsion usually during most of the flight.

A rocket straying off course could either smash into city under full power (thus being a cruise missile) or have its propulsion cut off remotely beforehand (thus being a ballistic missile), depending on a scenario. What kind of a correction were you making?

Cruise missiles are designed to rely on aerodynamic lift (and typically use air-breathing engines, although there have been some "boost-glide" weapons that used rocket engines).
Insurance isn't magic pixie dust, it just moved the cost around.
If the organization underwriting the insurance is implicitly and explicitly insured (too big to fail) by the US govt, it might as well be magic pixie dust!
That really just highlights how scary fiat currency is.
They do the tests to validate that this rocket won't turn into a cruise missile, headed for the nearest residential area. Not an impossible scenario. Lots of energy stored up in one of those rockets

This is a big plot point in the "Twin Spica" series.

Could they not have mocked the payload?
The question nobody is answering....
Mocking a six ton payload is actually fairly expensive, to get the cg, vibration etc. right. A lot more difficult than payload = mock(satellite); NASA did it for the Hubble but so the ground crew could practice operations and not drop the billion dollar space observatory, not to try and save some money if something blew up during testing.
Surely still much cheaper than a two hundred million dollar satellite?
And at what frequency would it save the payload? In 85-95% of cases, payload is safely delivered to space, so the dummy just slows things down. In most of the remaining cases the payload is lost on it's actual ride into space (e.g. the last time SpaceX had a problem, CRS-7) where a dummy can't help. There have been a handful of pad accidents involving loss of payload, but they are very much the exception to the rule about the exception to the rule.
It is possible to damage the rocket and systems when mating the payload in the faring, which would not become apparent without further testing and somewhat negates the purpose of this test itself.
The Apollo 1 accident[0] was during a similar test (a "plugs out test", which tests the spacecraft independent of all the support systems on earth to verify that it works correctly on its own). The rationale behind all this tests is "better have a problem on the ground, where we can fix it, than have the spacecraft fail in space".

If you get interest in this stuff, the HBO miniseries "From the Earth to the Moon" has an episode about Apollo1 (and the series as a whole, though slightly dated on the FX side, is amazing).

[0] https://en.wikipedia.org/wiki/Apollo_1

I haven't seen it in a long time, but +1 on the recommendation. I need to go borrow the DVDs from my parents and watch it again.
The series is awesome, but the book on which it's based is even better: Andrew Chaikin's "A Man On The Moon".
My understanding is that it is up to the customer whether they want to integrate the payload before or after the hot fire test. Integrating before means higher chance of an incident like this, while integrating after means significantly longer turnaround between the hot fire and launch.
Thanks, that's the answer I was looking for. Other commenters seemed to think I was asking why they test at all!
Well, I know that In Space, No One Can Hear You Scream.
Because losing the payload amortized by the chance of its loss is less expensive than the 100% dead expense of a test rocket.
Per @SpaceX, the issue was not with the rocket itself, but a pad anomaly and the Amos-6 payload was to be Facebook's first-ever satellite.
Testing in production. Tisk tisk.
Every rocket is test fired before launch. SpaceX does do many things that might be considered "testing in production", but this isn't one of them.
This disaster may have bigger implications for Spacecom. They were in the middle of being acquired by Beijing Xinwei and the successful launch of Amos 6 was supposed to be a factor in closing the deal[1]. I have no idea what blowing up on the pad means for the merger.

1. http://www.satellitetoday.com/technology/2016/08/24/spacecom...

It's a deal breaker most likely, the terms of the deal were based on AMOS-6 becoming operational by march 2017.

There has been high level of criticism in Israel about the deal as many seen this as giving up it's sovereignty over its space platforms even tho under the deal the all operations of spacecom platforms would be still handled from Israel proper, I'm sure the r/conspiracies is already booming with sabotage theories.

But yeah it's not great for spacecom in general, Facebook also will now withdraw from their contract with them for supplying internet to Africa as AMOS-6 was supposed to replace AMOS-2 and enhance the coverage over Africa and the Middle East. https://upload.wikimedia.org/wikipedia/commons/e/eb/Amos-6-K... Israel is one of the few countries that is pushing satellite coverage in the area, AMOS-6 was supposed to grant up to 20gbps of bandwidth to Africa over the leased Ka band which was desperately needed.

Won't they just relaunch in a few months? Not sure what the rebuild timeline is.
No, they don't have redundancies for satellites. It's 2 to 3 years.
It takes years to build a satellite, the bus, and every component is purposely built each component and the satellite itself goes through months of testing and even if they could build one within 6 months all of the contracts would expire by then which will require Spacecom to open the bids again.

All of the contracts were based on AMOS-6 being operational by a given date, AMOS-6 has been "under construction" for probably 4-5 years, SpaceX was selected as the launch contractor in late 2012 originally for a mid 2015 launch but the launch was delayed for various reasons.

Would it go faster the second time around, since they've already validated the design?
The design might be the "easy" part, these are 6 tons of one off highly specialized components with a huge lead time. Each component is custom and built for order by a contractor the lead time on some of them can be years simply because there aren't that many companies that design and build space systems. That combined with the fact that they need to go to each subcontractor and ask them to bid again after Spacecom finds a way to finance it (there is no way the insurance payout is anywhere near the actual cost of the satellite) would probably mean that the time it takes to build a new one isn't that different than it was to build the original.

Think of it like this if you total your McLaren P1-GTR there isn't an easy way to get a new one ;)

> there is no way the insurance payout is anywhere near the actual cost of the satellite

Why not?

I'm 15 years removed from the TV biz, but I vaguely remember 'birds' costing like $350m to launch, something like $200m + for the hardware, then $150m for launch fees and misc pre-launch insurances. In some cases, I want to say they insured them before taking possession of them, like if some guy at Lockheed has a bad day and drops a wrench on exactly the wrong part, you're looking at a substantial delay which could substantially impact business. Companies would talk up these satellites to investors for years before they launched, hard to imagine them not being completely insured up in to space. Then there was launch insurance and I think it included operations in many cases, TV birds had a lifetime of like 7 years and the operational insurance paid out if they had a certain amount of failure before the lifetime was up, transponders die and they need to spend fuel periodically to keep them in place when the fuel is gone the bird is done. I want to say the launch and ops insurances were like another $100m to $150m. All told you were in for half a billion to get in in to place and that's not including the lease on the slot of sky... I'm pretty sure Echostar was involved in some lawsuits about it and there are probably some public details that came out of that; maybe they wanted to claim more damages than the insurer wanted to cover or something like that and then no insurers wanted to sell them operational insurance.

It's not cheap but if that's your business you absolutely insure it. It's not cheap though. Would investors let you get away not insuring one? Rockets not making it in to space isn't uncommon.

there are some cases were you can't insure, like Ariane 501.
That's why you always build two.
"That's why you always build two."

Unless you're a Raman. Then you'll build three.

Preppers often say, you don't have it, unless you have two of it, and you don't have a backup until you have three.
Sometimes it is better to have one, just a very good one. Depends on the budget.
They all say that so that people overstock useful items, so when nuclear zombie virus apocalypse hits and normals all die, there's lot of supplies stockpiled and easily accessible for the survivors.

s/

That's a maxim for computer data backups as well.
This was my first thought but then again I don't know how expensive the one was and if they built one and it launched successfully, what would they do with the second one? Seems like a damned-if-you-do-and-damned-if-you-don't scenarios.
The payload was reported at $200 million.

Although whether it would cost an additional $200 million to build a second is another question, but I doubt there would be that significant an economy of scale for just one extra unit.

Why does a satellite cost so much? I always thought it was a computer with solar panels orbiting in space using gravitational force?
I think a lot of it is using electronics that are less susceptible to radiation along with needing a clean room to actually build the thing. Not sure if that accounts for the entire $200 million.
> That's why you always build two.

They very seldom do, at least for non-constellation projects. Cheaper just to pay the insurance premiums for loss of hardware and loss of earnings.

Unless you're banking on the sat being operational in a specific time frame for an acquisition to go through, like Spacecom did. They probably lacked enough capital to do so. Plus the failure still takes time to figure out.
For things like the Mars Rover, they built two so that if something goes wrong, they have one in the warehouse that they can experiment with before they try to remotely repair the one on Mars. I don't know if people do that for Earth-orbit satellites, though.
Why does every satellite have to be custom made like this? Without knowing anything about it, it seems as though most communications satellites do pretty much the same thing?
Superficially maybe, but you want to send zero weight you don't need into orbit so it makes sense to build to the exact mission profile vs trying for a generalized solution.
> Without knowing anything about it

Sometimes it's wise to stop, read what you just wrote, then hit "back" rather than post a comment.

If I don't know something, it's far better to acknowledge that and ask than to just hide in ignorance.
They aren't built in very large numbers. There are maybe a couple dozen commsat launches per year, and they do have fairly different requirements a lot of the time. There are variables like frequency, spot size, transmission and receive power, and whether the satellite does any on-board processing or routing or whether it just acts like a "bent pipe." There is some standardization, but not a whole lot. Economies of scale just don't quite kick in enough yet.
There are many standard components, though satellites are low-enough volume that almost nothing is really "off the shelf". The custom bits and integration are plenty of work (and time) in themselves.
Also, we really don't want to have all the communication satellites to have the same bus, and find out years later that they have some bug that causes them to malfunction after years in orbit.
To a degree every satellite is custom because of custom requirements. To the extent that they're not they can use an existing platform, like Boeing does with the 702.
I admire Musk as much as the next guy but SpaceX has always seemed fraught in the QA department[1]. I'd be surprised if they can survive long enough to accomplish their Mars mission if they keep screwing up this often.

[1] Which also seems to be a problem at Tesla as of late.

Sounds like SpaceX just killed multiple deals for this company. Could someone who knows about this kind of stuff (I assume this subset of the population is small) comment on whether there is recourse against SpaceX for losing the payload?
Unfortunately accidents happen, I have a feeling that some insurance company somewhere is going to end up paying for the damages...
It's possible to insure against business risk as well as asset risk, though it's usually not worth the premium. Though I've never been in the space business I would be surprised if it were not common for launches.
Private launches are required to carry insurance in order to receive a license; I looked it up when the Orbital rocket had a similar RUD, and the government had mandated something like $140 million or so between government property contingency and general liability, if I recall.

Edit: Yeah, here's Orbital's (I can't find SpaceX's): https://www.faa.gov/about/office_org/headquarters_offices/as... -- last two pages. It looks like they revised this license recently for their return to flight, too?

The risk of this happening was almost certainly accounted for in the contractual relationship.
Quite possibly with a termination clause. Insurance typically covers the cost of the satellite, not the cost of delaying time to revenue by what could end up being several months or years.
For crying out loud, have you never heard of insurance? All satellite launches are insured and contingencies like this are allowed for in contracts because rocketry is an inherently risky endeavor. I know HN is a tech rather than a business community but the concept of insurance is an elementary one and something every literate adult should be aware of.
There is zero need for your tone towards a user asking a simple question. He didn't know, he asked in a nice manner and it was an innocent question. How do you expect someone to learn or be comfortably asking questions on HN if they're told they aren't a "literate adult" for not knowing the specifics of insurance coverage within a high risk business sector.
Not all launches are insured. One of the largest satellite TV companies in America claims to have launched their first satellite without insurance because they were so strapped for cash.
Insurance was only for post ignition, there will be no insurance payout for this accident.
Elon was talking of SpaceX's insurance. Read: SpaceX won't be recovering the cost of the launch vehicle or the payload.

Spacecomm likely has their own insurance. Hopefully more details on this will emerge.

This is false. Pre-launch accidents are covered by marine cargo insurance.
So sabotage could be a factor?
(comment deleted)
This is in the realm of conspiracy theory. No evidence points to this. Nothing the parent comment said suggests it. I think you are filling in the blanks with your own hypotheses but they're accusatory shots in the dark at this point. It's possible, sure, but so are a lot of things.
This waving of tinfoil is making me wonder just how vulnerable a rocket like this would be to a sniper rifle bullet. I mean, it's pretty big, and it's full of all sorts of explosive goodies. I'd expect the chances of something going wrong if it's hit by a shot just as it lights up for a test run to be rather high.

Edit: Now imagine if last week you'd shorted as much stock as you could in the companies involved...

I like the way you think. How could someone determine if this occurred? Rocket sabotage seems like a rather perfect crime, very little evidence is left afterwards.
Your legal and technical claims sound highly questionable to me. Satellite operators are not responsible for the successful operation of the launch vehicle, and one would expect their contractual arrangements to reflect that. That would be like firing someone because their commuter train derailed and they were late to work as a result.
Why did they make a contract contingent on factors outside the contractant's influence? that seems counterintuitive.
Based on what others have posted, it's not necessarily a clause like, "If you launch successfully, we go ahead with the deal", but rather a clause like, "If the satellite is operational within X amount of time, we will go ahead with the deal."

This explosion probably put them way past the X amount of time so the deal would fall through.

It's irrational to argue that you would pull out of a merger because a 3rd party's rocket failure delayed a launch. Contracts almost invariably include escape clauses to avoid liability for matters that are outside the control of the parties. If the launch was already scheduled at the time of the merger agreement (which seems likely, a rocket launch is not the sort of thing you arrange at the last minute) then Beijing Xinwei would be expected to factor the risks into its pricing calculations.
"It's irrational to argue that you would pull out of a merger because a 3rd party's rocket failure delayed a launch."

Maybe, maybe not. A lot of things intertwined - moreover, M&A is not the most rational area of business.

Also - one side may 'cancel', huff and puff about it, then use it as a premise to come back to the table and negotiate a much better deal.

The failure of the launch changes the real world situation. It's not irrational to argue that such a change also changes the pros/cons of a merger. The fact that it was related to a 3rd party is irrelevant.
Stock is down about 10%. Makes me wonder if high-speed traders seek realtime video feeds of such launches. Is the "official" feed delayed?
There was no official feed, it wasn't a launch, it was a pre-launch test fire.
I don't think the USG will allow SpaceX to be acquired by a Chinese company.
How stupid, to test with satalite on-board?? esp considering how expensive they are, they should use a dummy the same size and weight. You can't test satalite functionality on the ground so a completly pointless risk to include it.
There are actually things to test in the integration between the rocket and satellite. It's not like the satellite is a dump piece of steel sitting on top of the rocket. The payload has its own computer, power systems, and propulsion. The interactions between that and the launching rocket / ground systems matter. Also mating the two is non-trivial, so you can push the static fire back closer to the launch if you are doing it with the payload on board.

A static fire without the payload on the rocket was already done in Texas several weeks ago. This is the final dress rehearsal before launch. I always thought of these tests as low risk, but I guess nothing in rocketry is low risk.

Would they do so if human cargo?
Note that a human doesn't "plug in" to the rest of the launch vehicle the way a satellite does.

I imagine they _would_ do these tests with the crew compartment attached and dummy human loads strapped in to the crew seats.

Actually they specifically plan to load astronauts before fueling begins. If they're in the capsule the launch escape system (LES)[1] can propel them away from any on-pad disaster; not so if they're still walking across the gantry.

That said, NASA has raised concerns about loading crew pre-fueling as part of the NASA-SpaceX Commercial Crew development contract (CCDev).

[1] Here's a video of the LES test for Dragon 2: https://www.youtube.com/watch?v=1_FXVjf46T8

What's sad about it? They have insurance, it's just hardware. Nobody died or was even injured, so it's merely an invoncenience.
There are engineers out there who actually care about their work, and put years into it. It's really most sad for them.
For the some of those engineers, its a loss of an Olympic gold...
Apparently one of the satellites was meant for internet connectivity in the developing world. Zuckerburg is very mad right now. Lost time is lost opportunity.
Everything about Internet.org went in a way contrary to what Mr Zuckerberg wanted.

The bourgeoisie of the developing world he wanted to help protested, the political class looked into constraining it by law, the first world hated it for misrepresenting the ideals of the Internet and going against hard-earned net neutrality laws, local Internet service providers backed off because of the negative publicity...

And the satellites he wanted to use to circumvent that went up in flames.

> They have insurance

Nope.

>Elon Musk has stated that because the rocket didn't intentionally ignite for launch, the loss of payload is not covered by launch insurance.

Lives were lost, just incrementally; person-hours in building things that will never add value.
Not completely; person-hours involved in building satellites, which are ultimately unique things, add a whole lot of experience to people involved in the building and handling, as well as for the organization. Sure it's not as much as a successful launch, but not totally wasted.
(comment deleted)
The payload was apparently on the rocket at the time[1]. A $200 million loss there, which sucks for the insurance company but better than losing a $2b custom government bird.

[1]https://twitter.com/SpaceflightNow/status/771352977315684352...

> A $200 million loss there, which sucks for the insurance company but better than losing a $2b custom government bird.

In fairness, when did the government last lose a bird? Doesn't the ULA have a '100%' success rate (where success is getting payload to space, though some payloads fail to make it to the planned orbit)

I suspect the $2b custom government bird refers to the space shuttle. Of which we have, in fact, lost more than one.
More like NSA/CIA/NRO spy satellites.
The shuttle was a vehicle with vastly different capabilities, and technical challenges to overcome, compared to a Falcon rocket.

There have been 133 successful shuttle launches, and 2 failed ones.

There have been 27 successful Falcon launches, and 2 failed ones.

All that was intended by my comment was to shed light on what the parent comment probably was intending to say.
And one of the Shuttle failures was actually on landing
> And one of the Shuttle failures was actually on landing

Reentry, not landing.

Interesting question of how you'd classify that.

A design flaw (foam) under launch stress resulted in foam shedding and leading edge wing impacts damaging tiles and ensuring loss of the vehicle on reentry.

I'd argue the failure was in design, triggered during launch, and manifested on reentry. But Columbia was doomed from seconds into flight.

The Falcon only has to solve the problem of getting the payload into orbit. The Shuttle had to both get payload into orbit and return its crew (and the Orbiter, and any return payload) safely to Earth (the Dragon capsule has to do this as well but the statistics given are for all Falcon launches). So as far as comparing the technical challenges involved with Falcon vs. the Shuttle, the Shuttle was solving a harder problem, which I think makes the numbers for the Shuttle look even better relative to the Falcon.

Harder at least as in there are extra steps that can go wrong (re-entry + landing), I suppose it’s debatable which was really a more difficult engineering project…

While true, NASA was able to have the shuttle program be so successful because of the things that they learned in the previous space programs. This is SpaceX's first space program. Don't compare a "senior" program to a "freshman" program.
Which is great advice - that should temper everyone's expectations of the firm (Which were sky-high a few days ago, when Musk was promoting his 'flight-tested' reused rockets.)
I think this is a false equivalence. Many if not most of the engineers working at SpaceX came from NASA or its constellation of contractors.
I know this will seem like nitpicking, but this was not a failed launch. It was during the inspection phase, which is performed to uncover flaws in the vehicle before launch. Small consolation I know because in this case the payload was lost anyway. But it's still important to note that the expectation of failure is going to be higher for inspection/static fire, and that in the case of commercial crew launches, the crew would not be near the pad at the time of static fire.
That figure's low by a factor of 100 for the Space Shuttle program.
Specifically referring to the cost of the orbiter vehicle itself (or the cost you'd incur building a replacement if you lost one), roughly $2 billion is a decently accurate figure. The shuttle program as a whole, yes, is drastically more expensive.

"The Space Shuttle Endeavour, the orbiter built to replace the Space Shuttle Challenger, cost approximately $1.7 billion." http://www.nasa.gov/centers/kennedy/about/information/shuttl...

ESA’s Ariane also has a far better successrate than SpaceX’s Falcon, and if I’d send a satellite to space, unless it would be a really cheap one, I’d likely not use the Falcon as delivery vehicle.
So you would rather pay $600 million per launch with 95% chance of success vs $200 million per launch with 93% chance of success?

There is no such thing a 100% chance of success, everyone has a chance of failure. I would not put ESA more than a few % more reliable. Math says your satellite would need to be really expensive for the price difference to be worth it.

Depending on the cost of failure, however for virtually every mission the 600M$ is a winner this is a no-brainer.

That "little" difference has huge huge economic repercussions, cost of insurance, ease of financing, and many other things. Not to mention that a satellite going boom can kill an operator if it doesn't have a backup. This isn't an insurance case where you get your money and try again the week after, building a satellite is a huge undertaking it takes years to build, years to negotiate the contracts for and you have to do it from scratch all over again if it goes boom.

The flip side is that estimating the reliability of a launch vehicle is an inexact science at best given the paucity of actual launches to provide data points (frankly, there was no real way of assessing the reliability of the Falcon 9 other than to note that launch failures or partial failures tend to occur more in the first couple of years of launches of a new rocket type is backed by a reasonable amount of evidence across the history of other launch vehicle programmes)
The satellite build costs $200m.

Launch 1 costs $600m: total cost, $800m.

Launch 2 costs $200m: you can build 2 spare satellites for the same cost (assuming you don't pay for the launches that go boom; and even if you do, that still gives you one spare satellite).

Production doesn't work like this all the time, not everyone is the US government.

Building 2 would cost more than double, or it will take twice as long.

There are huge lead times involved, subcontractors with other commitments and other bottlenecks in the pipeline. Not to mention that even if you do manage to build 2 in the same time window at only double the cost you still have the issue with the launch window, a launch accident would most likely ground the provider for a very long time which means you have to find another launch provider.

Spacecom had a launch window from spacex, spacex has a queue of 9-10 more launches already set, there will now be a huge delay and possible cancelation even if Spacecom had another unit they couldn't just launch it next month.

Given SpaceX's long manifest of scheduled missions, it seems like the people who actually pay for this stuff don't always agree with you. Going with the more expensive option is apparently not a "no-brainer."
Where did you take your $600 million and $200 million cost per launch? Ariane costs are around $95 million, but for bigger payloads than what the Falcon 9 can handle.
Someone already answered about the price, regarding the success rate, there is a much bigger difference than this:

Space X has had 29 launches, 2 failures and 1 partial failure (F9-004) so a success rate of 89.6%.

Ariane on the other hand hasn't had a single failure or partial failure since 2002 so if you compare since Space X started its services it has a success rate of 100%. If you take the whole history of Ariane it has indeed a success rate of 95.2% or 95.4% for the Ariane 5 version but the fact that Ariane has flown 72 consecutive missions without failure (2002 - Present) compared to 14 for Space X (2012 - 2015) says a lot about their reliability.

One has to wonder why it was thought to be a good idea to keep an expensive payload onboard during a live-fire test

EDIT: Did some research, it seems that those tests (without the payload) already ran successfully, and that this accident happened during the fueling stage at T-3 minutes... that's super unlucky :/

This wasn't launch. It was still a live fire test, but as you say, there are previous tests before integration.