The nice thing about smokeping is that a smokeping targets file and its whole configuration under /etc/smokeping/* can be very easily moved between places and set up somewhere new in less than 10 minutes. And its charting of the output of fping does a remarkably good job for such a simple tool.
For best results set your smokeping to run on 60s intervals not 300s.
Thanks for sharing this, I had no idea that existed as a thing. I had for a while thought about making something like this, running on some cloud somewhere, pinging home - but looks like I don't need to now.
Agreed, they basically launch Starlink satellites for free as they can reuse launch vehicles other customers already paid off.
Remains to be seen whether this is intentional or not. Given how a few of the satellites had a much sharper decay in orbit I would lean towards the latter. Doesn't seem to make sense that they would take them up only to bring them down again if they were setting them at a lower orbit.
> they basically launch Starlink satellites for free as they can reuse launch vehicles other customers already paid off.
Yes and no.
First, their customers actually prefer to use previously-flown boosters now. Insurance premiums on launches are slightly higher for first flights because the booster is 'unproven'. But more importantly, there's still big costs involved. The entire second stage of the rocket is expensive to build and it's not reused. Refurbishing the boosters takes time, people, effort.
Still cheaper than any of their competitors, and they get it "at cost", but we're still in the tens of millions range.
The US, Russia, Europe and China have burned many multiples of SpaceX expenses to find their space programs obsolete. That is what losses look like.
SpaceX spent a fraction of their development expenses, and now sells launch services profitably for a fraction of their costs.
SpaceX has been a wealth generating dynamo.
Raised estimated $9.9 Billion [1]
Worth based on recent funding $127 Billion [2]
NASA gave them a boost, with the result that NASA is going to save billions as they continue shifting launches to SpaceX. (Cancelling SLS would result in even more savings.)
Any perception of overall economic "losses" for SpaceX, its investors, or its customer-partners, comes from incomplete accounting.
Do you have a source for the insurance premiums being higher? I really doubt that is true but Spacex may offer a discount for previously-flown boosters which acts as an incentive.
The satellites are a kind of solution to the queuing theory problem. Queues work better when a small proportion of the tasks are very high priority, and a large number of very low priority tasks are available to pack in around them.
Not to pile on but as an addition to what others have said:
>as they can reuse launch vehicles
Falcon 9 is not fully reusable, "only" the first stage is. That was still an incredible accomplishment, and still saves a great deal of money because the first stage is the largest and has the most engines, which are one of the more complex and expensive parts of any rocket. However, the second stage is expended, so that's a fixed cost each time, IIRC estimates are around $9m. Even the fuel isn't nothing as other costs start to get brought down, F9 burns tens of thousands of gallons of RP-1 per launch and RP-1 isn't free either (probably at least $17/gal, so that's probably at least another half mil). Refurbishment of course but also recovery ships, salaries, launch pad time, it all adds up. Yes by historic scales this is all pretty cheap and SpaceX is certainly large enough with enough cadence, revenue and profit to absorb losses fine, they very intentionally run hardware rich. In a careful manner it should be made clear, they aim to learn things, but they do push to find the edges of the envelope on occasion as well. But it's not remotely free. Don't forget, there is also the lost time and cost of the actual sat capacity, they do actively want these things and are pursuing a cut down while waiting for Starship for a reason. They need more network capacity for Starlink. Losing a bunch of V2 Minis is another pile of monthly revenue they won't be getting for a while longer.
The Starship system will be a big step forward in all respects, the first rocket engineered from the ground up around economics. Fully reusable and more easily, cheap to construct, even the fuel is cheap. But even there they still have expensive things to be careful with, they've gotten more cautious recently because losing ground facilities in an explosion at this point would genuinely be very painful and set them back months which itself would represent major opportunity cost. And they can't simply spread things out because of the limits of the geography they're working with right now.
It's all still rocket science. Pushing hard in a controlled manner is good, they've done incredibly putting together a system around that and it's paid off. But not being reckless, and major losses are still going to sting.
At a much smaller and less costly scale than Starship, Stoke Space (Kent, WA) has a plan and has published CGI videos of a proposed fully reusable second stage. Along with something very similar to a Falcon 9 resuable first stage. Whether they are actually able to do it remains to be seen.
It doesn't matter if SpaceX gets discounted cost from reused rockets and not taking profit, because SpaceX is losing out on money they could get from selling launch. There is backlog of payloads for Falcon 9 so there opportunity cost for internal launch. The effective cost for Starlink launch is the $67 million that others pay.
I'm not sure I agree. These aren't satellites like the GPS satellites where each one cost hundreds of millions or billions to make, let alone launch. These are more akin to fancy cube sats where you can mass produce and make them (relatively) cheap.
Still very expensive, but if a company was structured around building lots of these then this wouldn't be game ending.
Cube sat is a bit of an understatement of the size and complexity of these satellites. IIRC they are roughly the size of a small car (albeit only a few hundred kilograms).
Yeah, the comparison would be okay-ish for the old ones, which where about 1m^3 in volume, just big and flat instead of cubes. The new V2 are ~5x the weight @2k8 pounds, the V2 Mini ~3x @1k8 pounds.
Finding out that there's a systematic issue with a new model that was overlooked until after launch might be, though. It does betray deficiencies in engineering and testing.
Then there was that incident where they launched a batch into the wrong kind of space weather, the atmosphere had been heated up by a solar flare, and the ion engine couldn't overcome the increased drag from the temporarily increased density. It looks wasteful.
In a way, yes. Many of the systems on these satellites rely on math and physics we've only recently mastered, high speed radio telecommunications and all that goes with it is truly magic, a modern marvel.
Inertial stuff however, that's all basic Newtonian physics, we've had that mastered for a bit longer.
Side note/fun fact - many satellites use tape measures for this purpose, literally tape measures. They're called gravity gradient booms.
So, I get your point, it's not a simple mistake compared to me forgetting my keys this morning, but it is a simple mistake in the context of a satellite.
my main point is that its silly to assume the satellite designers at SpaceX made some kind of remedial physics error as opposed to a more subtle error that is the result of engineering or design tradeoffs
It's a bit like a keel for a (water) ship. You lower it and the gravitational pull on it will be slightly higher down where it is than up where the satellite is. Or you raise it up above where the satellite is -- same idea.
The "tape measure" is a mass at the end of a tether (the "tape") and they are of course not "literally tape measures".
They are literally tape measures. Source: I know engineers at Lockheed martin..
Manufacing that tape is a highly specialized big factory scale production process. It doesn't make sense to build a special tape factory just for the 100 feet of it they need for satalites annually. The tape many of them use is sourced from the same suppliers as tape measures for these reasons.
One of the qualities of aerospace is that we get less forgiving of people making the same mistake over time. If someone mixed up metric and imperial units on a mars lander again they’d be vilified.
Yes, a basic mistake. I'm not sure what you're getting at with your comment, but this is literally the kind of thing that should be caught very early on, probably just by updating a few parameters (or even just one) in the attitude control sim and observing the outputs.
If this claim is true (a big "if", admittedly), this is a breakdown in the kind of fundamental systems engineering that they teach to undergrads building cubesats.
Same type of mistake as Ariane 5. Upgraded the size of some parts (from Ariane 4), but failed to thoroughly check every other system component would be compatible with the up-scaling. Specifically: the real-time control software was carefully written for the small parts, but not adequate for
the upgraded parts because the bigger numbers would overflow a fixed-width integer field.
- "The greater values of BH caused a data conversion from a 64-bit floating point number to a 16-bit signed integer value to overflow and cause a hardware exception.[5] The programmers had protected only four out of seven critical variables against overflow to keep within a required maximum workload target of 80% for the on-board Inertial Reference System computer, and relied on assumptions which were correct for the Ariane 4, but not Ariane 5, trajectory about the possible range of values for the three unprotected variables.[6]"
The complaints I’ve heard is that LEO is great but there’s a carrying capacity for any orbit and people are worried that starlink is over that limit even for LEO. I’m not orbital mechanics expert but when we discussed it at the time I believe it was agreed that a collision can launch debris into higher orbits but on a parabolic path, so if it doesn’t encounter anything else in a couple orbits it probably never will.
Also some of the LEO SpaceX has been approved for are higher than the ISS, not just lower.
Kessler is mostly worried about long term debris patterns, but one that cleans itself up in a few months could still cause a many billions in damage and loss of life. Also starlink enables some terrestrial activities that might not be wise without it. A cascade could leave some earthbound people fucked as well. That’s partially on them of course, but the optics are not great.
All starlink satellites are LEO, just varying degrees of it. They release the satellites from the second stage deliberately quite low, because it's more efficient in total weight carried by second stage, and more efficient in specific impulse to spend weight in kilograms of propellant tankage on the satellite for ion/hall effect type thrusters (specific impulses of 2000 ISP and greater) than to place N additional kg of fuel on the second stage to release them all in a higher orbit.
At the release altitude they're above the vast majority of the earth's atmosphere and solar panels + ion thruster burning prograde are sufficient to raise their altitudes to the intended operational orbit.
>... as long as their Kessler Syndrome half-life is low enough. And, you know, you haven't launched thousands of them.
It's hard not to sigh at the persistence of this getting applied mindlessly. The quantity isn't particularly important here, altitude is. There is still atmosphere at hundreds of kilometers, and thus there is still drag and in turn orbital decay. It falls off non-linearly with increasing altitude, but for low LEO and VLEO is very significant. You can see from the graph shown in the linked tweets that these only got to around 380km. At those altitudes lifetimes are measured in months [0]. You may also find it handy for visualizing to look at it in animated form with a Gabbard diagram [1], same result. Debris aren't an issue at low orbits because of how fast they decay, regardless of numbers. Indeed making use of that is directly part of the reason why SpaceX does Starlink launches the way it does, inserting low and then raising gradually via sat thrusters rather then just aiming directly for final orbit. Yes there are some reuse energy advantages and such, but it's also so that if there is a major problem (like this, or the batch they lost from unexpected solar atmospheric heating increasing drag) they fail safe, naturally decaying rapidly and burning back up even in the case of total control loss.
I'm curious what you two thought I meant when I said "Kessler half-life".
Did you not think I was talking about decay?
There's also two ways to think about it. What's the chance that we make a permanent wall of debris, and what's the odds that we take out everything in LEO and then have to wait months to get back to or from ISS.
>I'm curious what you two thought I meant when I said "Kessler half-life".
Since that's not really a term and you don't really seem to demonstrate any understanding here I don't know. You now claim to be talking about decay, except then you still say stuff that makes no sense if you understood decay, orbital inclinations, the kinds of distances involved here, or even where the constellation under discussion actually exists at all.
>What's the chance that we make a permanent wall of debris
At 380km with a few dozen sats? Zero.
and what's the odds that we take out everything in LEO and then have to wait months to get back to or from ISS.
Also zero, though this is nonsensical all over since ISS is at ~415km. What "Kessler" mechanism are you proposing to "take out everything" from 400-2000km (LEO) across all orbits, and somehow this doesn't hurt ISS at all (since your statement necessarily implies ISS still exists to go to/from)? And also prevents us from getting to/from 415km, which could only be done by incredibly dense debris below that. Space is big, this is stuff spread out over a greater area then the surface area of the Earth, massive debris could be an issue for something that wanted to stay in an orbit but odds of anything hitting a spacecraft merely moving through a single altitude in a matter of minutes/hours is minuscule even before ground based tracking. How is this big field of debris forming at <400km and not immediately starting to drop out? How does it even get self-sustaining in the first place given speed of decay there? In fact how many satellites do you think exist in all the orbits <400km, let alone in ISS' specific orbit? You talked about "thousands" with extra emphasis, so does that mean you think that Starlink maybe is at 380km as its permanent deployed orbit rather then the 550-570km that is its actual deployed orbit? Do you think that just maybe ISS has in fact been considered by nations worldwide and perhaps there are rules and extra caution around its orbit? Some things to contemplate upon, or even research.
James Webb has something like 300 things that had to go right for it to work. I thought, “that’s nuts” but apparently they thought it could be done and here we are. But that may also explain why it went way over the time estimates.
They also lost something like 40 out of 52 satellites when a solar flare caused the atmosphere to temporarily expand, causing more drag than expected before they could all go into thrusting prograde with their ion/hall effect thruster for standard post-release orbit raising. Immediately after launch.
At the scale of 3500+ satellites and automated assembly line production of them this is nothing more than a minor inconvenience. It's unfortunate that this happened but also it's great that if this is in fact the problem with the hardware design of the satellites it was revealed on the very first launch of the "v2.0 mini" production line. I'm sure they're pushing it into a "v2.0 mini rev B1" now or something in Redmond.
Anyone care to recap for those of us who'd rather not pull up twitter? The graph linked to in other comments appears to show a deliberate de-orbiting, or at least lowering of orbit.
Edit: There appear to be a few people, at least, who think this comment is inappropriate or not-in-the-spirit-of-HN. Fair 'nuff; that would be my opinion also if I thought the asker was being lazy. For the record, I attempted but was unable to find any relevant info by searching for such as "starlink v2 satellite" or "tmfassociates" (the last post on their blog was mid-Feb.) For now, at least, the only source appears to be that twitter post (thread?) The comments here are just informative enough to be maddeningly cryptic ("mind-blowing basic mistake"?)
I think there’s a general trend online where people post tweets on places like Reddit where others might assume the tweet is factual. As a standard disclaimer, just because someone claimed on Twitter that SpaceX messed up the reaction wheels does not mean it’s true. Unless there’s more concrete proof, we should not automatically assume it’s true.
74 comments
[ 2.0 ms ] story [ 69.5 ms ] threadhttps://www.w6rz.net/starlinkv2.jpg
And it took absolute ages for industry to get to the level where those graphs could be recreated easily in something else.
Even now it's pretty hard to recreate something like what https://oss.oetiker.ch/smokeping/ does with it.
For best results set your smokeping to run on 60s intervals not 300s.
I've been using it for years now
For SpaceX it will be an mildly expensive lesson.
Remains to be seen whether this is intentional or not. Given how a few of the satellites had a much sharper decay in orbit I would lean towards the latter. Doesn't seem to make sense that they would take them up only to bring them down again if they were setting them at a lower orbit.
Yes and no.
First, their customers actually prefer to use previously-flown boosters now. Insurance premiums on launches are slightly higher for first flights because the booster is 'unproven'. But more importantly, there's still big costs involved. The entire second stage of the rocket is expensive to build and it's not reused. Refurbishing the boosters takes time, people, effort.
Still cheaper than any of their competitors, and they get it "at cost", but we're still in the tens of millions range.
This is hilarious, and also an incredible testament to the work SpaceX has done in moving the rocket industry forward.
I see in the article that SpaceX is proudly carrying on their tradition.
SpaceX spent a fraction of their development expenses, and now sells launch services profitably for a fraction of their costs.
SpaceX has been a wealth generating dynamo.
NASA gave them a boost, with the result that NASA is going to save billions as they continue shifting launches to SpaceX. (Cancelling SLS would result in even more savings.)Any perception of overall economic "losses" for SpaceX, its investors, or its customer-partners, comes from incomplete accounting.
[1] https://www.crunchbase.com/organization/space-exploration-te...
[2] https://spaceexplored.com/2022/05/23/how-much-is-spacex-wort...
Launches are never free. Cheaper maybe, but free is a vast overstatement of what SpaceX is doing.
>as they can reuse launch vehicles
Falcon 9 is not fully reusable, "only" the first stage is. That was still an incredible accomplishment, and still saves a great deal of money because the first stage is the largest and has the most engines, which are one of the more complex and expensive parts of any rocket. However, the second stage is expended, so that's a fixed cost each time, IIRC estimates are around $9m. Even the fuel isn't nothing as other costs start to get brought down, F9 burns tens of thousands of gallons of RP-1 per launch and RP-1 isn't free either (probably at least $17/gal, so that's probably at least another half mil). Refurbishment of course but also recovery ships, salaries, launch pad time, it all adds up. Yes by historic scales this is all pretty cheap and SpaceX is certainly large enough with enough cadence, revenue and profit to absorb losses fine, they very intentionally run hardware rich. In a careful manner it should be made clear, they aim to learn things, but they do push to find the edges of the envelope on occasion as well. But it's not remotely free. Don't forget, there is also the lost time and cost of the actual sat capacity, they do actively want these things and are pursuing a cut down while waiting for Starship for a reason. They need more network capacity for Starlink. Losing a bunch of V2 Minis is another pile of monthly revenue they won't be getting for a while longer.
The Starship system will be a big step forward in all respects, the first rocket engineered from the ground up around economics. Fully reusable and more easily, cheap to construct, even the fuel is cheap. But even there they still have expensive things to be careful with, they've gotten more cautious recently because losing ground facilities in an explosion at this point would genuinely be very painful and set them back months which itself would represent major opportunity cost. And they can't simply spread things out because of the limits of the geography they're working with right now.
It's all still rocket science. Pushing hard in a controlled manner is good, they've done incredibly putting together a system around that and it's paid off. But not being reckless, and major losses are still going to sting.
Still very expensive, but if a company was structured around building lots of these then this wouldn't be game ending.
Losing a handful of satellites that are produced by the thousands in a factory is not quite as big a setback as it may seem.
Then there was that incident where they launched a batch into the wrong kind of space weather, the atmosphere had been heated up by a solar flare, and the ion engine couldn't overcome the increased drag from the temporarily increased density. It looks wasteful.
Inertial stuff however, that's all basic Newtonian physics, we've had that mastered for a bit longer.
Side note/fun fact - many satellites use tape measures for this purpose, literally tape measures. They're called gravity gradient booms.
So, I get your point, it's not a simple mistake compared to me forgetting my keys this morning, but it is a simple mistake in the context of a satellite.
Could you elaborate a bit? What part of a satellite is a tape measure useful for? How is it useful?
* https://tigersats.princeton.edu/cubesatkit
An unrolled tape measure is stiff so that the satellite/weight system will align up and down. As skinny things tend to do in orbit.
The "tape measure" is a mass at the end of a tether (the "tape") and they are of course not "literally tape measures".
https://en.wikipedia.org/wiki/Gravity-gradient_stabilization
Manufacing that tape is a highly specialized big factory scale production process. It doesn't make sense to build a special tape factory just for the 100 feet of it they need for satalites annually. The tape many of them use is sourced from the same suppliers as tape measures for these reasons.
Yes, a basic mistake. I'm not sure what you're getting at with your comment, but this is literally the kind of thing that should be caught very early on, probably just by updating a few parameters (or even just one) in the attitude control sim and observing the outputs.
If this claim is true (a big "if", admittedly), this is a breakdown in the kind of fundamental systems engineering that they teach to undergrads building cubesats.
https://en.wikipedia.org/wiki/Ariane_flight_V88?useskin=vect...
- "The greater values of BH caused a data conversion from a 64-bit floating point number to a 16-bit signed integer value to overflow and cause a hardware exception.[5] The programmers had protected only four out of seven critical variables against overflow to keep within a required maximum workload target of 80% for the on-board Inertial Reference System computer, and relied on assumptions which were correct for the Ariane 4, but not Ariane 5, trajectory about the possible range of values for the three unprotected variables.[6]"
When satellites are cheap and launching then is cheap, this might be a reasonable approach.
Oh. Oops.
Also some of the LEO SpaceX has been approved for are higher than the ISS, not just lower.
Kessler is mostly worried about long term debris patterns, but one that cleans itself up in a few months could still cause a many billions in damage and loss of life. Also starlink enables some terrestrial activities that might not be wise without it. A cascade could leave some earthbound people fucked as well. That’s partially on them of course, but the optics are not great.
At the release altitude they're above the vast majority of the earth's atmosphere and solar panels + ion thruster burning prograde are sufficient to raise their altitudes to the intended operational orbit.
It's hard not to sigh at the persistence of this getting applied mindlessly. The quantity isn't particularly important here, altitude is. There is still atmosphere at hundreds of kilometers, and thus there is still drag and in turn orbital decay. It falls off non-linearly with increasing altitude, but for low LEO and VLEO is very significant. You can see from the graph shown in the linked tweets that these only got to around 380km. At those altitudes lifetimes are measured in months [0]. You may also find it handy for visualizing to look at it in animated form with a Gabbard diagram [1], same result. Debris aren't an issue at low orbits because of how fast they decay, regardless of numbers. Indeed making use of that is directly part of the reason why SpaceX does Starlink launches the way it does, inserting low and then raising gradually via sat thrusters rather then just aiming directly for final orbit. Yes there are some reuse energy advantages and such, but it's also so that if there is a major problem (like this, or the batch they lost from unexpected solar atmospheric heating increasing drag) they fail safe, naturally decaying rapidly and burning back up even in the case of total control loss.
----
0: https://www.spaceacademy.net.au/watch/debris/orblife.htm
1: https://old.reddit.com/r/space/comments/ld4vlq/gabbard_diagr...
Did you not think I was talking about decay?
There's also two ways to think about it. What's the chance that we make a permanent wall of debris, and what's the odds that we take out everything in LEO and then have to wait months to get back to or from ISS.
Since that's not really a term and you don't really seem to demonstrate any understanding here I don't know. You now claim to be talking about decay, except then you still say stuff that makes no sense if you understood decay, orbital inclinations, the kinds of distances involved here, or even where the constellation under discussion actually exists at all.
>What's the chance that we make a permanent wall of debris
At 380km with a few dozen sats? Zero.
and what's the odds that we take out everything in LEO and then have to wait months to get back to or from ISS.
Also zero, though this is nonsensical all over since ISS is at ~415km. What "Kessler" mechanism are you proposing to "take out everything" from 400-2000km (LEO) across all orbits, and somehow this doesn't hurt ISS at all (since your statement necessarily implies ISS still exists to go to/from)? And also prevents us from getting to/from 415km, which could only be done by incredibly dense debris below that. Space is big, this is stuff spread out over a greater area then the surface area of the Earth, massive debris could be an issue for something that wanted to stay in an orbit but odds of anything hitting a spacecraft merely moving through a single altitude in a matter of minutes/hours is minuscule even before ground based tracking. How is this big field of debris forming at <400km and not immediately starting to drop out? How does it even get self-sustaining in the first place given speed of decay there? In fact how many satellites do you think exist in all the orbits <400km, let alone in ISS' specific orbit? You talked about "thousands" with extra emphasis, so does that mean you think that Starlink maybe is at 380km as its permanent deployed orbit rather then the 550-570km that is its actual deployed orbit? Do you think that just maybe ISS has in fact been considered by nations worldwide and perhaps there are rules and extra caution around its orbit? Some things to contemplate upon, or even research.
James Webb has something like 300 things that had to go right for it to work. I thought, “that’s nuts” but apparently they thought it could be done and here we are. But that may also explain why it went way over the time estimates.
> It remains to be seen if there will be other problems that take longer to discover
Which is a comment that applies to literally everything that humans have ever done in all of history.
Not only is this a safe area, they appear to be deorbiting and they'll be gone very soon.
> And, you know, you haven't launched thousands of them.
They have only launched a handful of the ones having trouble.
> Oh. Oops.
?
https://www.google.com/search?client=firefox-b-d&q=starlink+...
At the scale of 3500+ satellites and automated assembly line production of them this is nothing more than a minor inconvenience. It's unfortunate that this happened but also it's great that if this is in fact the problem with the hardware design of the satellites it was revealed on the very first launch of the "v2.0 mini" production line. I'm sure they're pushing it into a "v2.0 mini rev B1" now or something in Redmond.
Edit: There appear to be a few people, at least, who think this comment is inappropriate or not-in-the-spirit-of-HN. Fair 'nuff; that would be my opinion also if I thought the asker was being lazy. For the record, I attempted but was unable to find any relevant info by searching for such as "starlink v2 satellite" or "tmfassociates" (the last post on their blog was mid-Feb.) For now, at least, the only source appears to be that twitter post (thread?) The comments here are just informative enough to be maddeningly cryptic ("mind-blowing basic mistake"?)
https://archive.is/gzNG0
And some level of "learning" is expected considering it's only the first launch of these.
It's not the entire V2 (or V2-Mini) program in trouble, just this batch.