If someone from SpaceX is reading (after 5pm, of course), can we get a more nitty-gritty of what's going on software-wise on your end? I'd love to see what control and embedded wizardry you guys are up to.
Yep, this and there is a Quora question somewhere I think iirc.
It's mostly LabView I think, which is an interesting choice given all the hate it gets.
For those that don't know, LabView is a graphical (drag and drop) language for the most part where you interconnect various hardware components together. The advantage is that a lot of labs need to plug a lot of devices and sensors to run an experiment and LabView has support for a large amount of hardware. So you connect it all and can graphically view what the outputs are doing in chart form. The downside is that I don't think it's always very easy to specify exactly what you want to do, and code can pretty easily become spaghetti. There are a lot of horror stories about engineers having to maintain someone else's code.
Edit:
-Flight software is C++
-Launch is mostly LabView and yes, they have LabView running in mission control
We use LabVIEW, probably one of the few non-aero startups that do. It's great for parallel computation and test stand stuff, but I doubt they're flying LabVIEW executables. Most of the utility of LabVIEW comes on the V&V side (running lots of tests and logging lots of data).
Also it's very easy to build maintainable code in LabVIEW (contrary to the popular narrative), but there's a serious lack of good learning resources, and most people don't even take Core 1 and 2 before trying to code. Kind of like how a person on the business end of things may discover VB.NET and create some horrifying macros, simply because they don't know any better.
I think the main advantage of LabVIEW is not the language but the hardware it runs on. While it is ungodly expensive it's very reliable in a wide range of temperature & vibration.
I used LabVIEW to code the control / data logging of a prototype racecar and the hardware only broke once when we probably zapped it with a nasty ESD or reverse polarity connection somewhere. We never figured it out and didn't ask when we sent the part to be repaired.
Programming in LabVIEW is something I wouldn't wish on my worst enemy though. It's extremely hard to debug and time consuming to refactor. I always felt that I was 10-50x slower than when I would code in a standard text-based programming language. Working collaboratively also put you in a world of pain because, at least when I used it and I think it's still the case, LabVIEW doesn't support git natively so you have to use an external tool. They have a tool to merge / diff their proprietary binary files but the UX is terrible and I think even if they really tried to do something good, diffing diagrams (which is what a LabVIEW program is IMO) is going to stay hard.
From my experience the people that code in LabVIEW are not software engineer and I had discussions where I was arguing that yes the channel feature is broken but that's not a reason to put all the variables used to communicate between components in a single global variable file for example.
But once you fought against the language long enough and have something that works you can be pretty sure it will keep working forever.
I'd disagree about debugging; you can create custom probes for any given wire that run arbitrary code whenever the data gets updated, that combined with the ability to "hold wire values" aka program state, after execution has finished (or even paused) is a godsend.
Re: merging; yes absolutely, this is totally broken. NI should either release the file format for VIs, or offer conversion to something saner like XML, which you could easily build a nice git-diff tool for.
I maintain that LabVIEW is a good language with a terrible teaching problem; you can do so much if you just master the queued message handler architecture, and you can do things in a way that are extendible and maintainable for years afterwards
How would I know? However, if you could think of that scenario, you can be 100% sure spacex thought of it over a decade ago and have measures to handle that situation
How about a response from SpaceX software guy that works on Dragon software, including these displays:
> Dragon is a fully autonomous vehicle, so it is capable of completing the trip to and from the ISS without any interaction from the crew. But the displays and button panel do provide the crew with capabilities should they need to take action due an unexpected scenario or emergency. As for the tablets and displays, the tablets themselves act as a sort of backup, and include copies of important data such as procedures. The displays themselves are designed to be fully redundant, so if a single display failed, the other 2 could fully take its place. Even if we were to experience a failure of all 3 displays, the crew has a button panel that can be used to initiate emergency responses, and ground commanding is also possible.
> -Jarrett
Mongo’s “return before write” behavior is what earned it this title. It makes it look better on benchmarks, convincing unwitting devs that it is a better choice.
Mongo and NoSQL, to me, are really symptoms of the Bootcamp craze that happened a few years ago.
It's way easier to understand than relational databases, can be simply browsed and retrieved as json (usefull when your only language is JS) and had good adapters for the cookie cutter frameworks used at the time. And you don't need to bother with a schema or joins.
There are use-cases for NoSQL but it's not a once size fits all solution.
Most of the standard is taken by the description of the standard library ; the language specs ends at page 457 in C++20 (last draft).
How long would be your language spec if every single function's documentation was part of it ?
To give a reference with other languages:
- This is a bit less than C# (whose standard language spec ends at page 462 in ECMA-364 2nd ed.).
- The python spec for 3.9.9 is 151 pages. Okay, but let's compare equivalent things in the C++ spec and Python spec: awaiting a coroutine.
* In Python (the whole "await" spec):
Suspend the execution of coroutine on an awaitable object. Can only be used inside a coroutine function.
await_expr ::= "await" primary
New in version 3.5.
* In C++ (5% of the "co_await" spec):
The co_await expression is used to suspend evaluation of a coroutine (9.5.4) while awaiting completion of the computation represented by the operand expression.
await-expression :
co_await cast-expression
But, for C++ this is only the first sentence (which is not written in an abbreviated style but as a proper sentence). Then a whole page explains the semantics in detail, and gives a complete example of how to use it. The python spec is just insufficient in comparison.
Interesting that they are doing updates at only 10 and 50Hz. I had kinda assumed they'd be looking at much higher frequencies for telemetry especially for things like the rocket motor metrics
I thought that too initially, but if you think about the physics it makes sense.. the larger the mass the more stable the object, the lower the frequency of acceleration needed to adjust it's trajectory. Also even though it's dynamic, it's not articulated or anything like dynamic robots.
Take the opposite, a little quad copter, very low mass, will need very high frequencies. I used to manually fly miniature remote control helicopters before all of the automatic control stuff came in, they were extremely, inherently unstable, very tricky to control.
Not really. Only when scaling mass, margin of error, and forgetting thrust/gravity.
And even if you scaled mass, thrust, and margin of error proportionally, you'd still have an object exactly as stable as before (force X applied to the small object gives you the same acceleration as proportionally scaled force Xs applied to a larger object). Gravity accelerates all things the same, and so do proportionally stronger thrusters.
But being off by one meter can destroy a small drone just as well as a large spacecraft, so you don't have the luxury of scaling your margin of error by significant amounts.
The only reason larger objects appear more stable to humans is because we, and the environment[1], tend to apply relatively weaker forces to them, and also you don't tend to notice they're already moving at 5m/s because of their size.
[1]Larger objects will better compensate for "truly random" forces, if their size causes them to encounter more of them, canceling them out and/or they're generally small.
You can still have a small but massive flying device and would end up with the same acceleration and jerk because need more thrust to zero the gravity out.
The bigger your thing is the relatively smaller the 9.8 m/s^2 is.
I found this also interesting. I can see how it simplifies the mathematics to use a fixed time interval.
Can anyone experienced comment if 10Hz is very different or the same as vehicle control systems such as a modern car?
Also, are these systems fixed or floating point? Fixed time intervals would help keep numerical instability of floats in check (however I'd really hope this is more formally understood for such a safety-critical system.)
I've been working on the engine control unit for trucks. I think the fastest control loops were 100Hz (but that was only a few), fixed time intervals. Then, of course, you have the fuel injection that is controlled by its own processor, independent of the ECU scheduling.
Nowadays I work with satellite SW. Most of the control loops are pretty slow. The fastest ones are those controlling gyros and reaction wheels that run in 5 or 10Hz
50Hz is pretty damn fast for anything happening at rocket ship scale (i.e. gimbaling of engines). 10Hz seems fine for a more strategic roll-up of all subsystems wherein you would have to do more complex logic each round. The physics are pretty stable at any rate considering the masses involved. If we were talking about controlling a 10g nanodrone (hypothetically), 50hz would probably be cause for concern.
For a SpaceX mission, there isn't a lot that can go wrong in 2 hundredths of a second that you could have magically resolved if you saw it 1 hundredth of a second earlier (i.e. at a 100hz rate).
> 50Hz is pretty damn fast for anything happening at rocket ship scale
I was thinking whether this is OK for dragon in-flight abort system. Well, probably is. Or that may use some system on Dragon itself as the rocket may be no more...
FTS/crewed abort is likely a special exception to these control loops. Last I read there are some lengths of wire that run down the rockets which, when broken, would allow for an instantaneous reaction to some catastrophic event.
I don't know what I'm talking about, but I believe in optimal control a rocket's orientation changes very slowly and the rocket is designed for aerodynamic stability. 50Hz is also probably much faster than the engine can gimbal or throttle. So if a 50Hz tick is too slow to respond to some stimulus, something has gone horribly wrong.
Telemetry can be packaged up and sent out at higher than the control loop rate (if needed). There's nothing keeping you from having a buffer that stores up higher frequency data that gets read off and sent off as a blob each control loop.
I would assume something like an engine controller or some other control loops might run internally at a much higher frequency. But the main control loop which orchestrates all components is able to run at lower frequency.
Keep in mind that this is _reaction time_. As in, if something happens to an engine how fast do I need to react to safe it before it spreads to other things. Turns out that 20ms response time is plenty appropriate.
Are there any processes in combustion dynamics faster than 20ms? You bet there are, and when the engine is being tested they are instrumented with high speed probes, because during development/testing you want to _understand_ what’s going. When you are flying you want to stay in control first and foremost.
Of course, it would be awesome to have everything on high speed data acquisition channels. But there is a very limited telemetry budget and everyone is fighting for it. So compromises have to be made.
There probably are high speed probes on the rocket, such as strain gages. And if an engineer thinks that having a high speed engine metric would be _critical_ to do postmortems then it will most likely be included.
So, I would say that assigning telemetry budgets it’s a holistic process where multiple teams fight/cooperate to share that budget and prioritize absolutely critical data channels.
Rocket engines have built-in computers, like ECUs in cars, that likely operate at a much finer update cycle than 50hz. Flight computers are not concerned with every aspect of timing when it comes to the engines, they just need to tell the engine's controller when to fire.
There's probably no point in going much faster for most control systems of physical devices of the sizes we're talking about. The actual response time of actuators and time to see the results of changes on sensors is much longer. Think the time required to actually adjust the throttle or gimbal a huge rocket engine, and then the time required to see an acceleration change as a result of that.
They probably use higher rates for stuff like measuring combustion dynamics during ground tests of the engines in development. If anything is wrong there, they'll probably seek to correct it via mechanical or fluid dynamics adjustments of the system itself, since active control systems could probably never react fast enough anyways.
These are almost certainly global control loops that are orchestrating hundreds or thousands of subordinate control systems that have completely independent dynamics and refresh rates.
It is fascinating that they solve robustly 50 times per second a control problem and handle gracefully numerical issues. Granted they have dedicated hardware to do so that fast, but it never stops to amaze me.
The data rate for flight control systems (air or space) is not that great but the requirement for reliability is. That’s not just an absence of software defects but also reliability in the face of hardware failures. (Compare the A320 flight envelope protection to the 737 MAX MCAS.)
In some ways, a quadrotor's control system is a more challenging controls problem! And I'm not at all suggesting that the rocket control loop isn't hard, but a few things to consider:
- Quadrotors control their attitude solely by adjusting the thrust generated by the four motors. Unlike rockets, or fixed-wing aircraft, or cars even, the moment there is any kind of disturbance (whether software, hardware, or environmental), the aircraft immediately starts to fall out of the sky.
- For the work I do, we typically fly around 5-7m above ground. If the motors completely quit, we have around 1.2 sec before we hit the ground at 42km/h.
- The aircraft is also capable of roll rates >50deg/sec. Even if you're high enough that you don't immediately hit the ground, motor problems can also result in you pointing upside down and pulling yourself towards the Earth instead of pushing away.
- Most of the parts have notoriously bad repeatability, plus small variations would wreck any repeatability anyway. You can't just set all of your motor throttles to 60% and expect them to have the same thrust, it's nested-upon-nested control loops to keep everything going
- The sensors are generally crap too with non-trivial amounts of bias and drift.
And the wildest part of all of this? You can get a top notch flight controller to handle all of this for you (and open source!) for $150 USD https://shop.holybro.com/pixhawk-5x_p1279.html?
> ...when you plan to reuse booster rockets and shuttles on future missions.
Shuttles? I'm aware of the general term shuttle as a vehicle for going back and forth, but here I'm certain that the better term would be craft, capsule, or Dragon. Certainly not shuttle.
No, I mean that they are talking about "boosters and ___" therefore ___ refers to a component, not the system as a whole. The term shuttle might refer to the whole system that is needed to shuttle people up and down, but the terms that fit the structure "boosters and ___" are craft, capsule, Dragon, maybe a few others, but not shuttle.
Doubly so when that is not a term that SpaceX uses to refer to any component or system.
Interesting they seem to ad-hoc the lifecycle processes, or at least no mention of any industry standards mandatory (RTCA/DO-178C, ISO26262 or even sys eng like SAE ARP4754, etc). Also, no talk of formal methods (formal modeling or formal verification, a la DO-333 with something like Simulink Design Verf or TLA+). Works for them obviously!
I’m not sure that I took this away from the article. Looked like they just didn’t go into detail. They mentioned storing data for keeping track of equipment health. To me that sounds like a super broad stroke that would be equivalent to historized time series data as well as contextualized asset data.
I’m not familiar with the standards you called out (not an aerospace engineer), but I would imagine it’s similar to what we do in the controls engineering and automation space.
I’m actually really curious about what software/methods they use to contextualize assets and what those data points are. How do They digitize the equipment data that comes from visual inspections/manual testing.
I am quite interested to see how they deal with regulations and standards (I'm in medical device space). I scanned through their job postings and I can't find a single standard listed (this is up to the Sr Engineer/Developer positions), as well as a lack of dedicated software QA/test positions.
Obviously they are doing some amazing, and quite high quality work, so I am quite interested in what their internal setup is like.
Yes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing).
It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent history. Quality should be meaningful and not just a rubber stamp.
Even in a self-regulated/peer quality control context, I think it's quite amazing to not have people tasked for dedicated (or more focused towards) testing and QA, even within the software/engineering group.
I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilities. One of the dangers of this setup is always that people (and their managers) prioritize output of their own areas, resulting in neglecting testing and review of other areas. It's certainly possible to make the setup, but it takes good people (and team cohesion/morale) up and down the chain to make this work.
Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time.
Prioritizing things is key, but at the end of the day there is a capacity to what one can do. It takes good management skills to realize the capacity of the employees and when things need to change. Luckily for me I’m working with groups in all three of the areas I was working in to offload those extra responsibilities to other people so I can focus on what I’m supposed to be doing (managing).
From seeing this work and not work at different companies, I've decided at root it's a company culture issue.
Specifically, "How does my company deal with failure?"
Which is another way of saying "Are people disincentivized from telling the truth at my company?"
You can never have a process that requires truth-to-power as a regular occurrence that's successful in an environment where there are career penalties to speaking uncomfortable truths.
That's why legacy providers & government have engineered a system that accepts +50% wasted time, in exchange for not requiring truth because almost everything is checked and verified.
It's been ~a decade since I was knowledgeable in SpaceX software development details, but at least then, they were not the kind of shop that emphasized rigor and formality. Their culture is oriented toward rapid implementation & test, pragmatism, and empiricism. That has its pluses and minuses.
My personal view is that more of the industry, including SpaceX, should be moving toward engineering things to be more correct-by-construction, including formal modeling & verification as you mention.
As one example, I don't hear most aerospace firms talking about SeL4 for their flight computer RTOSes. For another, you might be surprised (frightened?) by how many vehicles implement their flight software as big single address space executables written in C (sometimes C++) despite running on modern superscalar processors with MMUs. Even static analysis in some of these codebases is relatively recent - they are heavily dependent on functional testing for safety.
Do you know more that would substantiate defunct-ness?
- Your comment also reinforces my point: if aerospace seeks to be more rigorous, then more in aerospace should be figuring out how to shore up development of SeL4. Instead, they continue with no vision as to what should one day replace things like VxWorks.
Are the properties that have been proven for seL4 particularly relevant for avionics systems?
I thought they mainly focussed on process isolation and security. A flight computer would be much more concerned about deterministic performance and real-time guarantees.
I know on the testing/V&V side, they use a mix of LabVIEW and Python, at least for some specific teams. Lots of automated (physical) tests are being written in both.
How are distributed realtime systems like a rocket is synchronized? Are there central timeservers and allocated transmission slots and timestamped query-response action messaging or is it just small individual realtime nodes making their own decisions?
I’ve been wondering about this for a year or so because I can’t seem to find one that matches my description, other than concept drawings on NIST 4D/RCS paper, but there got to be one in the world.
iirc all flight systems are in triplicate and all data is checked against the others, then a quarum is needed from the 3 to make a decision.
other rockets use rad hardened computers that make decisions on their own, but for cost and time, I believe SpaceX just triplicated all necessary systems and votes out and reboots any system that gets bit flips.
They actually use more than three now. The basic architecture is that computers are organized in "strings". Each string is doubled. If the two don't agree, the result is chucked, and the system reboots. Each of these "strings" then participates in a quorum. The result is extreme reliability, with relatively fast convergence.
There's typically a central time source sending a 1Hz pulse into the main computer(s). The sensors will use something like Time Triggered Ethernet, RS-485, or a MIL-1553 bus.
Yeah this is in line with my experience. However there are concerns about the accuracy of quartz clocks as temperatures fluctuate, this poses a problem for some orbital patterns.
Thanks to both answers - I've asked couple engineers of my generation and "that would be an interesting concept" was the best response I got. I have so much to learn!
For LEO missions, the source is now typically a GNSS receiver outputting a PPS line. Otherwise, you try to have a precise clock (TCXO, OCXO, and now CSAC are available) as a reference distributing the time to all the subsystems, and you correct the satellite time offset from the ground (similar to NTP).
Yes. This does require all of the network devices between the nodes to also implement PTP (otherwise you fall back to NTP-like behavior) but this isn't a problem if you're greenfielding the implementation.
This article provides an excellent summary of why the defense and space industry is different from “tech”. What SpaceX is doing is normal practice in Defense, but it reads like “1975” when you’re used to TypeScript and cloud instances.
I really love my role of spanning the gulf between these worlds. There’s so much to learn from each side to the other.
Ada/SPARK is pretty close to unheard of in the US.
People tend to solve problems with the languages they know and can find people to hire who know it. C++ is fast, and there's a good number of people to hire who know it.
Actual comments I've heard from senior developers when I tell them I write Ada in my spare time:
- "That language is still around?"
- "Other than you, I've never heard or anyone who has ever used Ada."
when i was first exposed to ada, early 1990's, the language was unusable. compile times where excessive (like hours). machines of that era were slow and had small memories (4MB was a common size, and more than 1 CPU was a luxury).
i must also admit, ada was hard. you had to do some mojo just to print out a value ("package int_io is new integer_io blah, blah). it had a standard of a couple of hundred pages that you kept on your desk. and you looked at it on a regular basis. and the compiler was picky (strongly typed).
companies dropped ada because they couldn't hire. C++ people were more available.
my opinion today is ada is relatively small, makes very fast code, and supports all the programming paradigms of C++ in a very sane way.
if the first ada compiler was "ada95" instead of "ada83", it would be very popular. it came out a decade before the world was ready.
Ada 2012 is what I learned, but I probably would have moved on if had learned Ada 95. They've done a fantastic job modernizing it, and the next version looks even better (Ada 2022).
> my opinion today is ada is relatively small, makes very fast code, and supports all the programming paradigms of C++ in a very sane way.
This is why I've been sticking with it. I also get pre/post conditions, a real module system (yes, I know C++20), and bounds checked arrays/strings. Easy built-in multitasking has me using concurrency a lot more often as well.
> companies dropped ada because they couldn't hire. C++ people were more available.
I work professionally in C++, and honestly think you could convert a C++ programmer into an Ada one in about a month or so, due to the conceptual similarities.
Ada/SPARK is pretty close to unheard of in the US.
I'm confused, I knew a woman in the early 2000s that worked out of Fort Dix in New Jersey, doing ADA development for
U.S. Army tanks. She told me that ADA was the standard language for military hardware. Can anyone confirm or deny this?
There was historically a push along these lines (Ada as official DoD language for the software on the systems it buys).
I don't have the whole story as to when and how this stopped being a requirement (1990s?), or if it was ever a true requirement, but in U.S. DoD aerospace today, Ada is uncommon. I'm not sure of any new systems developed over the past 20 years that are based on Ada. Perhaps some of the avionics suppliers like Honeywell or Rockwell Collins that historically used Ada still do, but I don't think most aerospace prime contractors do today.
The prevalent U.S. Government view is that the industrial entities that implement systems should have the latitude to decide what they want to use. For DoD systems, they would just be expected to have the DoD conduct reviews of their software/system design. [Sadly, such reviews vary wildly in their quality and how much companies' feet are held to any fire.]
Keep in mind the FAA requirements for civil aircraft certification are different from DoD's airworthiness assessments. And then launch vehicles and spacecraft are assessed/certified on a different basis than aircraft.
I think they have a lead or two who were most comfortable doing it the way they like and this is all about it. If you have good software people just step away and do not tell them how to do their jobs.
A bit of a disappointing read. All I got out of it is that the individual control systems run at 10 or 50 Hz (less than I expected).
From my own background knowledge I remember that the rockets actually run on Intel chips and the Dragon touchscreens are Chromium-based (less certain about that one).
The control loop reminds me of how you would create a game ie. get inputs, update states, based on the current state of the environment and inputs, render (write output).
111 comments
[ 0.23 ms ] story [ 178 ms ] threadIt's mostly LabView I think, which is an interesting choice given all the hate it gets.
For those that don't know, LabView is a graphical (drag and drop) language for the most part where you interconnect various hardware components together. The advantage is that a lot of labs need to plug a lot of devices and sensors to run an experiment and LabView has support for a large amount of hardware. So you connect it all and can graphically view what the outputs are doing in chart form. The downside is that I don't think it's always very easy to specify exactly what you want to do, and code can pretty easily become spaghetti. There are a lot of horror stories about engineers having to maintain someone else's code.
Edit: -Flight software is C++ -Launch is mostly LabView and yes, they have LabView running in mission control
Also it's very easy to build maintainable code in LabVIEW (contrary to the popular narrative), but there's a serious lack of good learning resources, and most people don't even take Core 1 and 2 before trying to code. Kind of like how a person on the business end of things may discover VB.NET and create some horrifying macros, simply because they don't know any better.
I used LabVIEW to code the control / data logging of a prototype racecar and the hardware only broke once when we probably zapped it with a nasty ESD or reverse polarity connection somewhere. We never figured it out and didn't ask when we sent the part to be repaired.
Programming in LabVIEW is something I wouldn't wish on my worst enemy though. It's extremely hard to debug and time consuming to refactor. I always felt that I was 10-50x slower than when I would code in a standard text-based programming language. Working collaboratively also put you in a world of pain because, at least when I used it and I think it's still the case, LabVIEW doesn't support git natively so you have to use an external tool. They have a tool to merge / diff their proprietary binary files but the UX is terrible and I think even if they really tried to do something good, diffing diagrams (which is what a LabVIEW program is IMO) is going to stay hard.
From my experience the people that code in LabVIEW are not software engineer and I had discussions where I was arguing that yes the channel feature is broken but that's not a reason to put all the variables used to communicate between components in a single global variable file for example.
But once you fought against the language long enough and have something that works you can be pretty sure it will keep working forever.
Re: merging; yes absolutely, this is totally broken. NI should either release the file format for VIs, or offer conversion to something saner like XML, which you could easily build a nice git-diff tool for.
I maintain that LabVIEW is a good language with a terrible teaching problem; you can do so much if you just master the queued message handler architecture, and you can do things in a way that are extendible and maintainable for years afterwards
https://twitter.com/tobi/status/1266867215325974528
> Dragon is a fully autonomous vehicle, so it is capable of completing the trip to and from the ISS without any interaction from the crew. But the displays and button panel do provide the crew with capabilities should they need to take action due an unexpected scenario or emergency. As for the tablets and displays, the tablets themselves act as a sort of backup, and include copies of important data such as procedures. The displays themselves are designed to be fully redundant, so if a single display failed, the other 2 could fully take its place. Even if we were to experience a failure of all 3 displays, the crew has a button panel that can be used to initiate emergency responses, and ground commanding is also possible. > -Jarrett
https://old.reddit.com/r/spacex/comments/ncj4vz/we_are_the_s...
Most of the bad things about MongoDB are actually bad things about the NoSQL fad, and that fad was huge around here a few years ago.
It's way easier to understand than relational databases, can be simply browsed and retrieved as json (usefull when your only language is JS) and had good adapters for the cookie cutter frameworks used at the time. And you don't need to bother with a schema or joins.
There are use-cases for NoSQL but it's not a once size fits all solution.
[1] https://www.iso.org/standard/79358.html
To give a reference with other languages:
- This is a bit less than C# (whose standard language spec ends at page 462 in ECMA-364 2nd ed.).
- The python spec for 3.9.9 is 151 pages. Okay, but let's compare equivalent things in the C++ spec and Python spec: awaiting a coroutine.
* In Python (the whole "await" spec):
* In C++ (5% of the "co_await" spec): But, for C++ this is only the first sentence (which is not written in an abbreviated style but as a proper sentence). Then a whole page explains the semantics in detail, and gives a complete example of how to use it. The python spec is just insufficient in comparison.Take the opposite, a little quad copter, very low mass, will need very high frequencies. I used to manually fly miniature remote control helicopters before all of the automatic control stuff came in, they were extremely, inherently unstable, very tricky to control.
Not really. Only when scaling mass, margin of error, and forgetting thrust/gravity.
And even if you scaled mass, thrust, and margin of error proportionally, you'd still have an object exactly as stable as before (force X applied to the small object gives you the same acceleration as proportionally scaled force Xs applied to a larger object). Gravity accelerates all things the same, and so do proportionally stronger thrusters.
But being off by one meter can destroy a small drone just as well as a large spacecraft, so you don't have the luxury of scaling your margin of error by significant amounts.
The only reason larger objects appear more stable to humans is because we, and the environment[1], tend to apply relatively weaker forces to them, and also you don't tend to notice they're already moving at 5m/s because of their size.
[1]Larger objects will better compensate for "truly random" forces, if their size causes them to encounter more of them, canceling them out and/or they're generally small.
You can still have a small but massive flying device and would end up with the same acceleration and jerk because need more thrust to zero the gravity out.
The bigger your thing is the relatively smaller the 9.8 m/s^2 is.
Can anyone experienced comment if 10Hz is very different or the same as vehicle control systems such as a modern car?
Also, are these systems fixed or floating point? Fixed time intervals would help keep numerical instability of floats in check (however I'd really hope this is more formally understood for such a safety-critical system.)
Nowadays I work with satellite SW. Most of the control loops are pretty slow. The fastest ones are those controlling gyros and reaction wheels that run in 5 or 10Hz
For a SpaceX mission, there isn't a lot that can go wrong in 2 hundredths of a second that you could have magically resolved if you saw it 1 hundredth of a second earlier (i.e. at a 100hz rate).
I was thinking whether this is OK for dragon in-flight abort system. Well, probably is. Or that may use some system on Dragon itself as the rocket may be no more...
Which is why the abort system probably doesn't need to wait for that tick.
Are there any processes in combustion dynamics faster than 20ms? You bet there are, and when the engine is being tested they are instrumented with high speed probes, because during development/testing you want to _understand_ what’s going. When you are flying you want to stay in control first and foremost.
Of course, it would be awesome to have everything on high speed data acquisition channels. But there is a very limited telemetry budget and everyone is fighting for it. So compromises have to be made.
There probably are high speed probes on the rocket, such as strain gages. And if an engineer thinks that having a high speed engine metric would be _critical_ to do postmortems then it will most likely be included.
So, I would say that assigning telemetry budgets it’s a holistic process where multiple teams fight/cooperate to share that budget and prioritize absolutely critical data channels.
They probably use higher rates for stuff like measuring combustion dynamics during ground tests of the engines in development. If anything is wrong there, they'll probably seek to correct it via mechanical or fluid dynamics adjustments of the system itself, since active control systems could probably never react fast enough anyways.
- Quadrotors control their attitude solely by adjusting the thrust generated by the four motors. Unlike rockets, or fixed-wing aircraft, or cars even, the moment there is any kind of disturbance (whether software, hardware, or environmental), the aircraft immediately starts to fall out of the sky.
- For the work I do, we typically fly around 5-7m above ground. If the motors completely quit, we have around 1.2 sec before we hit the ground at 42km/h.
- The aircraft is also capable of roll rates >50deg/sec. Even if you're high enough that you don't immediately hit the ground, motor problems can also result in you pointing upside down and pulling yourself towards the Earth instead of pushing away.
- Most of the parts have notoriously bad repeatability, plus small variations would wreck any repeatability anyway. You can't just set all of your motor throttles to 60% and expect them to have the same thrust, it's nested-upon-nested control loops to keep everything going
- The sensors are generally crap too with non-trivial amounts of bias and drift.
And the wildest part of all of this? You can get a top notch flight controller to handle all of this for you (and open source!) for $150 USD https://shop.holybro.com/pixhawk-5x_p1279.html?
You mean the shuttle has been replaced by something newer and hence is an obsolete word?
Doubly so when that is not a term that SpaceX uses to refer to any component or system.
I’m not familiar with the standards you called out (not an aerospace engineer), but I would imagine it’s similar to what we do in the controls engineering and automation space.
I’m actually really curious about what software/methods they use to contextualize assets and what those data points are. How do They digitize the equipment data that comes from visual inspections/manual testing.
I wish it was a little more detailed in general.
Obviously they are doing some amazing, and quite high quality work, so I am quite interested in what their internal setup is like.
It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent history. Quality should be meaningful and not just a rubber stamp.
It’s a fine line
I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilities. One of the dangers of this setup is always that people (and their managers) prioritize output of their own areas, resulting in neglecting testing and review of other areas. It's certainly possible to make the setup, but it takes good people (and team cohesion/morale) up and down the chain to make this work.
I definitely agree that it's a fine line.
Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time.
Prioritizing things is key, but at the end of the day there is a capacity to what one can do. It takes good management skills to realize the capacity of the employees and when things need to change. Luckily for me I’m working with groups in all three of the areas I was working in to offload those extra responsibilities to other people so I can focus on what I’m supposed to be doing (managing).
It would be interesting to see how they operate.
Specifically, "How does my company deal with failure?"
Which is another way of saying "Are people disincentivized from telling the truth at my company?"
You can never have a process that requires truth-to-power as a regular occurrence that's successful in an environment where there are career penalties to speaking uncomfortable truths.
That's why legacy providers & government have engineered a system that accepts +50% wasted time, in exchange for not requiring truth because almost everything is checked and verified.
Good culture: We failed. I succeeded.
Bad culture: You failed. I succeeded.
My personal view is that more of the industry, including SpaceX, should be moving toward engineering things to be more correct-by-construction, including formal modeling & verification as you mention.
As one example, I don't hear most aerospace firms talking about SeL4 for their flight computer RTOSes. For another, you might be surprised (frightened?) by how many vehicles implement their flight software as big single address space executables written in C (sometimes C++) despite running on modern superscalar processors with MMUs. Even static analysis in some of these codebases is relatively recent - they are heavily dependent on functional testing for safety.
Also: https://sel4.systems/news/2021
Do you know more that would substantiate defunct-ness?
- Your comment also reinforces my point: if aerospace seeks to be more rigorous, then more in aerospace should be figuring out how to shore up development of SeL4. Instead, they continue with no vision as to what should one day replace things like VxWorks.
I thought they mainly focussed on process isolation and security. A flight computer would be much more concerned about deterministic performance and real-time guarantees.
I’ve been wondering about this for a year or so because I can’t seem to find one that matches my description, other than concept drawings on NIST 4D/RCS paper, but there got to be one in the world.
other rockets use rad hardened computers that make decisions on their own, but for cost and time, I believe SpaceX just triplicated all necessary systems and votes out and reboots any system that gets bit flips.
[0] https://en.m.wikipedia.org/wiki/Precision_Time_Protocol
https://nasa.github.io/fprime/Architecture/FPrimeArchitectur...
I really love my role of spanning the gulf between these worlds. There’s so much to learn from each side to the other.
I was also reminded of James Gosling’s VJUG talk about the Liquid Robotics onboard software for autonomous ocean vehicles:
https://youtu.be/EgUctRxaiLs
People tend to solve problems with the languages they know and can find people to hire who know it. C++ is fast, and there's a good number of people to hire who know it.
Actual comments I've heard from senior developers when I tell them I write Ada in my spare time:
- "That language is still around?"
- "Other than you, I've never heard or anyone who has ever used Ada."
i must also admit, ada was hard. you had to do some mojo just to print out a value ("package int_io is new integer_io blah, blah). it had a standard of a couple of hundred pages that you kept on your desk. and you looked at it on a regular basis. and the compiler was picky (strongly typed).
companies dropped ada because they couldn't hire. C++ people were more available.
my opinion today is ada is relatively small, makes very fast code, and supports all the programming paradigms of C++ in a very sane way.
if the first ada compiler was "ada95" instead of "ada83", it would be very popular. it came out a decade before the world was ready.
> my opinion today is ada is relatively small, makes very fast code, and supports all the programming paradigms of C++ in a very sane way.
This is why I've been sticking with it. I also get pre/post conditions, a real module system (yes, I know C++20), and bounds checked arrays/strings. Easy built-in multitasking has me using concurrency a lot more often as well.
> companies dropped ada because they couldn't hire. C++ people were more available.
I work professionally in C++, and honestly think you could convert a C++ programmer into an Ada one in about a month or so, due to the conceptual similarities.
I'm confused, I knew a woman in the early 2000s that worked out of Fort Dix in New Jersey, doing ADA development for U.S. Army tanks. She told me that ADA was the standard language for military hardware. Can anyone confirm or deny this?
I don't have the whole story as to when and how this stopped being a requirement (1990s?), or if it was ever a true requirement, but in U.S. DoD aerospace today, Ada is uncommon. I'm not sure of any new systems developed over the past 20 years that are based on Ada. Perhaps some of the avionics suppliers like Honeywell or Rockwell Collins that historically used Ada still do, but I don't think most aerospace prime contractors do today.
The prevalent U.S. Government view is that the industrial entities that implement systems should have the latitude to decide what they want to use. For DoD systems, they would just be expected to have the DoD conduct reviews of their software/system design. [Sadly, such reviews vary wildly in their quality and how much companies' feet are held to any fire.]
Keep in mind the FAA requirements for civil aircraft certification are different from DoD's airworthiness assessments. And then launch vehicles and spacecraft are assessed/certified on a different basis than aircraft.
Exploring the software that flies SpaceX rockets and starships - https://news.ycombinator.com/item?id=27115372 - May 2021 (112 comments)
Also related:
SpaceX Software Team AMA - https://news.ycombinator.com/item?id=23444738 - June 2020 (6 comments)
We are the SpaceX software team, ask us anything - https://news.ycombinator.com/item?id=23432996 - June 2020 (28 comments)
Software Engineering Within SpaceX - https://news.ycombinator.com/item?id=23403800 - June 2020 (313 comments)
We Are SpaceX Software Engineers - https://news.ycombinator.com/item?id=13705762 - Feb 2017 (6 comments)
From my own background knowledge I remember that the rockets actually run on Intel chips and the Dragon touchscreens are Chromium-based (less certain about that one).
I was hoping for a lot more details like that.
No wonder John Carmack got into rockets!