I’m running an automotive firmware project at the moment and there are such things as software integrity levels, and each time you add something to the stack you create quite a large amount of testing and validation and process that needs to be followed.
And it’s pretty much impossible to take something existing like Chromium and certify it to what you need. The architecture is wrong. The process is wrong. There is a lack of formal requirement traceability. Many other issues.
But mixed-criticality systems are quite normal in many deployments nowadays, especially with the advent of virtualization and hypervisors you should be able to run safety critical and non-safety critical functions at the same time on the same SoC.
Yes, but you have to prove that if one display considered to be "non critical" shows misleading data, it will not cause harm to the crew. For example, if the Chromium crashed and showed the heading or fuel level incorrectly, would they fly the rocket ship into the ocean or run out of fuel and drop out of the sky.
Yes, it's called software integrity decomposition, and it's complicated and requires much analysis.
I believe the choice of Chromium and JS for the UI simply shows that the astronauts are merely inconvenient payloads. Gagarin's first flight was already fully automated 59 years ago.
Should've paired them with a dog. As the old joke goes, the astronauts are there to feed the dog; the dog is there to bite the astronauts if either of them tries to touch any of the controls.
As we saw in a recent Space Force episode on Netflix, the ape was much better than the dog. But you need pair them, as with Doug and Bob. Doug being the ape of course.
The UI design consisting exclusively of elegant but low contrast shades of blue certainly gives a similar vibe. But that might still be misleading, perhaps they deliberately keep the routine visualizations in a low key "style over substance" passenger entertainment mode so that they have plenty of visual headroom for when something actually important combs up?
That's an interesting thought. If the UI is always blue-ish...and suddenly a bright red triangle appears with an Exclamation Point inside....you might be about to die and should act accordingly.
This reminds me of the Airbus concept. They follow what's called a "lights out" design for all of those buttons on the overhead panel. When things are set and working the way they should be for normal flight, there is no indication. Only if something is non-normal does the indication or button light up.
For example, buttons that should be "on" for flight only light up when the system they control is "off", and things like emergency overrides or backup systems that should be "off" normally have buttons that light up when they're "on".
It's an interesting concept. Obvious benefit is you can glance at the overhead panel and instead of having to consider the state of, say, 50 or more buttons, you can instantly see if everything is "normal" for flight. But the downside is that it's not as intuitive to answer the question "is system X turned on (or off)"...you have to look at the button, then know what's considered "normal" for that system, on or off, and determine if the lack or presence of indicator light means on or off. That said...the pilots know the plane well enough where it's not a problem.
Right, I think the crew dragon is intended to fly fully automatic from launch to docking? But imagine if there is an emergency and the crew need to take over, but lo and behold, chromium crashed! Hopefully the spacecraft can still be operated without the display.
I think it is not helpful to call out that their architecture is "wrong" and their process is "wrong".
The attribute "wrong" is very opaque.
As somebody who doesn't have your background, i cannot follow you. I have no idea about automotive and space software requirements.
Since there are many aspects that lead to choices of a system architecture it requires more words to explain why a choice is not a good choice for the job.
I was referring to the architecture of Chromium being wrong to achieve certification for safety critical applications.
And look, I haven't said impossible, but rough estimate I'd say removing 75% of the features and 5+ years to rewrite it to be compliant. And the subset of javascript that it could run would be very limited - it might not even have loops or functions.
That was the most surprising part to me too, but not that surprising. Lots of automotive and newer industrial UIs use Qt Quick with QML which is not all that different from HTML5 and JS with something like React.
It's used in the presentation layer, not the actually control systems.
You might need multibit ECC also in CPU caches. Pretty much all CPU caches have ECC, but no idea how large of an error they can fix.
Then again, probably control logic is running on something radiation hardened and GUI layer (with some redundant hardware) can be easily rebooted as necessary.
I don't think that's necessarily true. I've worked with a couple of cacheful CPUs that only used parity checks on caches, not ECC. To get safety, the cache has to be set up as write-through instead of write-back, so that it is almost always safe to throw away a line that suffers a parity error.
Interesting that it’s three identical processors running he same software. I remember just the automated Vienna central train station used a pair of machines, x86 and SPARC, one running the logic in a procedural language (Chill) and the other running in a logic language (Prolog). They didn’t want both processors to share any code, and therefore potentially the same bug.
That hasn't been the case in Vienna for some years, so you must have seen that setup a decade or so, ago.
These days its THALES-built systems running SIL-4 rated code in a best 2-of-3 or 3-of-5 voting system, and each of the nodes can be either PPC, Arm or x86 - but not all the same, for architecture reasons. These systems produce a vote that has to match, or else the system shuts itself down. Chill and Prolog haven't been shipped in that capacity in decades...
(Disclaimer: worked for Thales on these systems, had to help remove a ton of Chill code, fix bugs in its compiler, etc.)
>They didn’t want both processors to share any code, and therefore potentially the same bug.
They're not worried about bugs, they're worried about electromagnetic radiation (which is way more present when you get outside the atmosphere) flipping bits.
There have been cases of homogenous CPU architectures in safety-critical where all CPU's failed at once.
Bitflippinrads are one of a multi-variate set of problems that these systems are designed to detect and avoid. Nodes can degrade otherwise: SRAM failure, battery/cap depletion, all sorts of things. The idea is to detect when a voting node has had a catastrophic system failure, and then remove offending nodes from the decision as rapidly as possible.
A multi-architecture system assists with managing (and certifying) things such as compiler bugs and impact from system-level homogeneity, where critical components all exhibit the same bug/failure. An x86 hardware bug is fatal if all 5 of the nodes in a voting system are x86 and making a critical decision; it is less of an issue if the other 2 or 3 nodes are PPC, ARM, MIPS, etc., or a mixture there-of, and - as variant hardware architectures - are democratically still voting during the period .. the idea is to avoid homogeneity and pursue heterogeneity.
They are worried about bugs, but they address those in other ways. This is some of the most extensively integration tested software there is (I was part of the team that built the integration testing). Every commit is ran through a custom CI system that successively tests pieces of the system working together up to and including automated hardware tests. The number of lines of code dedicated to testing the vehicle is definitely much greater than the lines of code on the vehicle.
This is exactly how the Airbus fly-by-wire system works as well. Two software stacks running on two separate hardware configs get bundled together in a single box. Then they take three of those systems and feed them from different redundant sensors together with pilot input and have the system vote on the output. If one computer returns a different result than the other two, it’s assumed to be faulty (or fed by faulty sensors) and no longer considered for input. On top of that there are actually separate redundant computer systems that run different parts of the plane with some overlap in case one system completely fails.
There are reports that the procedure for fixing bugs on the STS was so heavy-weight that they often redefined non-fatal bugs as correct operation. The example I read of was an error event that dumped thousands of lines of text to the flight console screen, obscuring useful information.
Basically every change, whether software or hardware, had a lengthy acceptance process. Reviews, analysis, verification and validation, documentation, operation and maintenance procedures, etc.
Heaven help them if the Javascript they run depends on npm components.
When I read about these multiply redundant systems I always want to know what provisions are made for restarting or shutting down a part that is producing wrong answers. Can the Power device reset the machine whose instructions lost the vote? If the two cores disagree, do they both reset?
Also, what is done for procedures that must not be interrupted? E.g., once de-orbit is initiated, if the microcontrollers get no new instructions, do they have enough to re-enter safely? Or do they need updates whenever something is supposed to happen?
Why is this comment grayed out... This is so true.
Assuming they dumb enough to put chromium on a spaceship for a UI then there a good chance that yes they did use bower/npm. Unless they designed it in pure HTML/JS... again this seems unlikely ... given the blunt stupid of the initial choice.
Sounds kinda disappointing/irresponsible. Chromium is an overly complex, relatively buggy piece of software. For example I recently experienced a bug where a text I selected was significantly offset from what should have been selected given the pointer position (on linux), a misclicky kind of bug. Considering that the majority of Crew Dragon's controls are through touch screen, maybe even manual docking(not sure, but I don't see a joystick anywhere), my personal experience with chromium does not instill confidence for such critical applications. But I guess they tested it very thoroughly.
C++ and linux do not strike me as mission critical piece of software / language either. My heart desires for something with verified properties, advanced languages with integrated proof checkers, verified OSs, or at least smaller purpose built OSs that were carefully reviewed. But I guess SpaceX is not to blame for this. It's likely such approaches are simply not ready yet. And I know that C++ and linux are common in mission critical applications. JSF avionics is written in C++ for example.
(Not to mention JavaScript in space, in a mission critical role. That shouldn't have happened.)
Once you decide that absolute reliability is unattainable, you can instead aim for sufficient redundancy. It’s not perfect, but with enough safe guards in place it may very well be good enough.
Software is not always like hardware though: the same software failure can cascade across redundant instances rapidly just as easily as it affects the primary.
The same hardware failure can cascade too; redundancy alone is not a panacea, you need some safeguards. Ideally, nothing should fully trust anything else.
Maybe cascade was the wrong word. I meant that if you pop in a hot spare and it's running the same software, there's a good chance it will encounter the same issue, sometimes immediately.
I was just reading that they design the rocket so that one engine could explode without affecting the others (hopefully) and they test the engines by dropping a nut or bolt or something in, which is not something rocket engines normally can tolerate.
I disagree. It is easy to build something presentable with HTML and browsers, and there have been many JS frameworks that lower the barrier. However, that is not because JS is a good language for it; it is because JS is the only language available in that environment.
In fact, there are at least two other languages that transpile to JS, because you need JS to be able to run in a browser. But JS the language is not good enough, so you want two write your code in something else.
Building a UI in a completely different language, without a browser, is definitely harder, but you might have several advantages in the other, non-UI parts.
You haven’t made any arguments about why Javascript isn’t good for building UIs there. Here’s mine for why it is:
- first class functions (callback style is a good fit for UI programming)
- async/await and single threaded - many components of UIs are async by nature, Javascript supports you well here while keeping things simple with its single threaded model
In the context of writing reliable software for a spacecraft? Pretty much anything with a reasonable type system, for a start. Maybe C#, or some reasonable subset of C++20, or Rust. Maybe Reason or OCaml.
I mean very clearly the good people at SpaceX have heard of these languages and decided against it. I think JS and Chromium are the best UI kit I can think of in terms of man hours to get X done. If you're careful the only thing I've ever had issues with is latency, but in space if you're in a situation where missing a frame or two is important you're already beyond fucked.
Many people are using TypeScript these days, it works very well.
I like C# and Rust too. I've built UI's in C++ and MFC/OpenGL/DirectX. It remains my fairly strong opinion that JavaScript is well suited to UI development.
In theory, maybe. In practice, your UI thread very rarely has to do the kind of heavy lifting that would block it so badly you actually have problems.
I have encountered this -- for example parsing very large incoming JSON payloads -- but very rarely. And if you really need to mitigate it, there are webworkers.
Javascript has also evolved quite a bit and there is also TypeScript that improves on that. The language is not perfect by any means, but it gets better.
Chromium is being used in the UI. Not on critical flight software.
> maybe even manual docking
It isn't done manually
> C++ and linux do not strike me as mission critical piece of software / language either
A lot of flight software is done on C++, as you even mentioned. In the end, your best reliability partner is not using a language that's only know by fewer people.
What would be a good "mission critical piece of software"? QNX (with no memory protection?). Linux is fine. I suppose they're not firing Ubuntu out of the box and running that with no customizations.
Separation of concerns and redundancy usually beat correctness on the long term (which doesn't mean they didn't test the correctness of the system).
Its probably a misunderstanding, but QNX has a very strong memory protection model that extends to its driver stack. Given that its a true realtime/microkernel, which has its own set of problems, but memory protection isn't one of them. For a number of cases QNX is going to be a large step up from linux.
Unlikely that they are using javascript in the actual flight controller. Also under the screen they have a small amount of manual buttons that is likely to be seperate from the touchscreen interface logic.
> I meant that in aerospace applications UI is mission critical too.
It is, but usability of the UI is mission critical as well. The humans in the capsule need to be able to quickly and effectively consume information in a way that doesn't make important facts hard to get at, and then quickly and effectively perform commands to address issues.
I don't think a touch GUI is a bad choice in this regard, and if I'm building a touch GUI I'd rather use some existing framework. My personal choice would've been something like Gtk or Qt though :)
In this case, C++ code is written in a different style than regular applications: exceptions are not allowed, allocations are not allowed past a certain point in program initialization, raw pointers are frowned upon, etc. This is common in this type of software and when you pair it with the right development culture it really makes a difference in the robustness of the result.
It is never persisted outside of RAM. The flight computers are never powered off during a mission. You rely on two flight computers to always be functioning, that’s why the system is one-fault redundant.
You can think as each string being “functional“: given the same set of inputs you expect the same set of outputs.
Are you asking about what happens if all the power goes out or are you asking how they get state back? If everything reboots it’s fine. On restart the computers will look at all the sensors add project the state of now. There is no reason to keep the state of the past is there? Why does it matter where the craft was 5 minutes before? What matters is what is what is in front of them and the future right? If you tell the computer I want to fly home it looks at where you are and tells you how to get there. It does not need to remember the past parts of the trip does it?
It'd have to know that it's destination was earth, or does it just sit idle after a power on and wait for the command? What happens if the astronauts are asleep? I feel like it needs some persisted data.
None if possible. In the safety critical parts you want to use software that
- don't do dynamic any memory allocation, has bounded memory usage, never runs out of memory.
- has bounded execution times
- has low interrupt latency
- can do checksum on data and internal state an option. (if need to load into memory)
- is tested using the most rigorous standards, 100% verification and code coverage.
If you write great relatively general but customizable database system like this with good indexing, you need only 5-10 customers and you are set. Safety-critical systems are growing in size and there is real need for software that can get easily certified.
SQLite could probably get certified as a part a system at least as read only database. It's relatively solid code. But it will cost big money and I don't think anyone has done it.
I'm reasonably sure that quite a bit of the complexity is more or less directly caused by the amount of features it provides. I agree with you in that Chromium probably brings in a lot of features they don't need, but I would also say that reimplementing the features they do need is also a huge liability and would incur quite a bit of complexity too... but it'd make it their insular system with no other resources flowing into improving it.
I don't have the data to make that call, but I would think someone at their end evaluated this quite thoroughly and determined the risks of rolling their own to exceed the risks of using Chromium.
(Also they mentioned on the stream that they do in fact have a few hardware buttons for important things. Can't cite this, sorry...)
Most safety critical code in the world, both civilian and military, is currently written in C/C++. Ada/SPARK is also used but it 's below 50%.
These languages have the best practical static code analyzer, verification and proofing tools money can by. I'm personally using Astree https://www.absint.com/astree/index.htm Automatic docking software for the ATV that delivers supply to ISS is written using C code and verified with Astree.
People get stuck into the language, but if it has some features that make it work, you can use it with verifiers. Then you write tests and simulators and tests ...
JavaScript and Chromium seems more suspect to me, but if the code is well tested and has limited run time, I think it can be good to go. The state is stored and prosessed in those radiation hardened RAD750 processors. Chromium/JavaScript runtime image can restart every few seconds if needed.
The UI and controls seem like unnecessarily flashy bullshit to me. Usually if you diverge from KISS principle bad things happen. But before I make any judgements I would see how they made those decisions.
In the case of SpaceX, it seems that they use embedded Linux and C++ with x86 processors, same as standard PC processors. A reason for using embedded linux is that, it allows using standard C++ or even scripting languages for controlling the hardware from the user-space by just reading and writing files. Linux device drivers (aka kernel modules) maps the hardware to special files on /sys or /dev. For instance, it makes possible to control a GPIO (General Purpose IO) which the device driver maps to sysfs special file system, by just writing 1 or 0 to the file a like /sys/class/gpio/gpio4/value which would enable GPIO 4 and turn on a LED attached to it. Another practical example about this feature is that, on Linux, it is possible to turn on or turn off the keyboard capslock LED by writing to some /sys/class/leds file such as "$ echo 1 > /sys/class/leds/input7\:\:capslock/brightness" which turns on the Capslock LED. By writing 0 to it, the LED is turned off.
This feature of Unix and Linux allows controlling the hardware with any programming language capable of reading and writing files, including Python and standard C++. I guess that they may be using standard PC hardware with industrial IO card. They may also use a single-board computer or custom board with x86 low-power variants processors built for embedded applications as SOC system-on-chip. One example of low power x86-SOC based processor is: https://www.cnx-software.com/2015/04/09/vortex86dx3-is-a-new...
The whole Dragon is a simulation written in Python and running inside a VM running inside an OS written in JS and running inside an Electron app running in a container somewhere on Amazon AWS.
From the latest video NASA uploaded, the astronauts explain that the UI sends "commands", and if it fails, they have physical buttons below the screen to send these commands.
The UI at 2:22 is very similar to the one discussed in https://news.ycombinator.com/item?id=23162820 as seems to use three.js and GSAP. Also, the open-source Lato font seems to be used throughout the interface.
That'd be a really nice touch, one of my favourite fonts for effects. Released after client who commissioned it, a chain of retail bookstores, rejected the design. Largest deployment till now, although just dumped in redesign, was for headings at classifieds website OLX. Caught eyes for a while I guess.
No need then for an especially legible font for cockpit screens like Airbus required https://b612-font.com/ following decades of practices in avionics. Of course the use-case is different and I'm sure everything is adequate for SpaceX goals. Like with the several layers of spontaneously rebooting virtual machines in Tesla cars, in the end branding looks gorgeous.
A slight inaccuracy: the flight strings are not x86, but ARM, running on a custom board.
Also, only the actual graphical display application uses Chromium/JS. The rest of the system is all C++. The display code has 100% test coverage, down to validation of graphical output (for example if you have a progress bar and you set it to X% the tests verify that it is actually drawn correctly).
> but I would not be surprised if you could power-cycle all displays if needed in the middle of a mission.
Indeed you can, judging by the instruction to power off and clean all displays (then power them back on) prior to ISS rendezvous/docking in the live stream.
SpaceX has had a booth at GDC at least two times, they attend in order to recruit game developers (programmers), as they feel these programmers work with a lot of the same constraints they do in terms of resources and experience with real-time, performant systems, etc.
For triple redundancy, is there a way to re-sync if one of them disagrees? Control systems tend to have internal states that may be disrupted even if an input has glitched for a single cycle. Will it eventually converge back to nominal? Or is that processor rejected for the duration of the mission?
If a flight string desyncs (i.e. it has state at the end of a cycle different from its two siblings) then it is automatically rebooted. Then state is automatically resynced from any other sibling after boot. There is no converging between the strings, they are either in complete sync or they aren’t.
Fascinating. The increased complexity in the flight control software and user interface probably makes the job of being an astronaut easier in the “happy path” - less cognitive load interpreting metrics and raw telemetry when you have the computer doing that for you.
My wonder then is how to effectively manage when things go wrong. The Apollo spacecraft were able to be essentially hot patched directly in the cockpit via the dsky. Can you do that with a fancy touch screen interface now? Perhaps up/down links to the capsule from earth are good/reliable enough to send patches from ground control, if need be? I wonder how the astronauts feel about a potential “loss” of control in case something goes wrong?
Depending on the platforms and customers C and Java are also heavily used next to Ada. The history behind Ada is mostly that the military required components to be made for that language. That requirement shifted around 10 years ago to a preference.
Probably running Node Javascript (or Typescript), transmitted in JSON over a restful API, hosted on a K8s cluster using Docker containers. Also, the JSON payload has a formal schema in the form of JSonSchema V5. /s
On a more serious note. Some of the amazing stuff that the SpaceX team are accomplishing really puts into perspective the day to day work I and others here do.
The Flight Software team is about 35 people. We write all the code for Falcon 9, Grasshopper, and Dragon applications; and do the core platform work, also on those vehicles; we also write simulation software; test the flight code; write the communications and analysis software, deployed in our ground stations. We also work in Mission Control to support active missions.
That's not just hilarious, that's also pretty close to how I feel most of the time.
I mean, some people have built the Airbus A380.
On the other hand, last week, I created a test DB with the user "test" and the password "test". I couldn't connect to it, and after a long debugging session, noticed I misspelled "test".
I appreciate it's a joke, kicking at open doors at that. Still, here's what one dude coded in JavaScript on his spare time a few years ago: https://www.pouet.net/prod.php?which=71881
You can get a lot done when you're not re-inventing HTML dropdown form elements or replacing the Android date picker because some "stakeholder" just really, really needs them to look on-brand. And other low-value bullshit tasks done solely on the guesswork-hunch-feelings of someone whose job is to create new Jira tasks. See also: 90s development teams who made all kinds of things you'd expect a team 5x as large and a timeline twice as long for, today.
Other references say parts of Falcon9 and Dragon a version of Unix calLed VxWorks from Charles River. It is realtime (not virtual) and predates Linux. NASA uses it in a lot of its space probes because its been debugged for 33 years.
That doesnt means its perfect. The two 2003 Mars rovers experienced a bug in the then-newfangled flash memory driver. It was successfully fixed and reloaded from Earth.
We used it for “hard realtime” embedded system didn’t even use virtual memory. Used to have pretty archaic toolchain (diab and some clunky eclipse setup). The os itself seemed pretty meh too (we had license and hence all the sources). Not sure how you would even get to try it without some trial agreement with their rep? Maybe try Qnx instead? It implements Posix so at least you’ll have easier time moving to linux. Also hard realtime and actually is a micro kernel arch which is nice
159 comments
[ 2.7 ms ] story [ 229 ms ] threadThis is the most interesting part for me being a web dev! How likely do you think this is?
I’m running an automotive firmware project at the moment and there are such things as software integrity levels, and each time you add something to the stack you create quite a large amount of testing and validation and process that needs to be followed.
And it’s pretty much impossible to take something existing like Chromium and certify it to what you need. The architecture is wrong. The process is wrong. There is a lack of formal requirement traceability. Many other issues.
Yes, it's called software integrity decomposition, and it's complicated and requires much analysis.
For example, buttons that should be "on" for flight only light up when the system they control is "off", and things like emergency overrides or backup systems that should be "off" normally have buttons that light up when they're "on".
It's an interesting concept. Obvious benefit is you can glance at the overhead panel and instead of having to consider the state of, say, 50 or more buttons, you can instantly see if everything is "normal" for flight. But the downside is that it's not as intuitive to answer the question "is system X turned on (or off)"...you have to look at the button, then know what's considered "normal" for that system, on or off, and determine if the lack or presence of indicator light means on or off. That said...the pilots know the plane well enough where it's not a problem.
The attribute "wrong" is very opaque.
As somebody who doesn't have your background, i cannot follow you. I have no idea about automotive and space software requirements.
Since there are many aspects that lead to choices of a system architecture it requires more words to explain why a choice is not a good choice for the job.
And look, I haven't said impossible, but rough estimate I'd say removing 75% of the features and 5+ years to rewrite it to be compliant. And the subset of javascript that it could run would be very limited - it might not even have loops or functions.
It's used in the presentation layer, not the actually control systems.
Then again, probably control logic is running on something radiation hardened and GUI layer (with some redundant hardware) can be easily rebooted as necessary.
These days its THALES-built systems running SIL-4 rated code in a best 2-of-3 or 3-of-5 voting system, and each of the nodes can be either PPC, Arm or x86 - but not all the same, for architecture reasons. These systems produce a vote that has to match, or else the system shuts itself down. Chill and Prolog haven't been shipped in that capacity in decades...
(Disclaimer: worked for Thales on these systems, had to help remove a ton of Chill code, fix bugs in its compiler, etc.)
Still, the basic point is interesting: the Thales system also uses a separation of architecture.
They're not worried about bugs, they're worried about electromagnetic radiation (which is way more present when you get outside the atmosphere) flipping bits.
Bitflippinrads are one of a multi-variate set of problems that these systems are designed to detect and avoid. Nodes can degrade otherwise: SRAM failure, battery/cap depletion, all sorts of things. The idea is to detect when a voting node has had a catastrophic system failure, and then remove offending nodes from the decision as rapidly as possible.
A multi-architecture system assists with managing (and certifying) things such as compiler bugs and impact from system-level homogeneity, where critical components all exhibit the same bug/failure. An x86 hardware bug is fatal if all 5 of the nodes in a voting system are x86 and making a critical decision; it is less of an issue if the other 2 or 3 nodes are PPC, ARM, MIPS, etc., or a mixture there-of, and - as variant hardware architectures - are democratically still voting during the period .. the idea is to avoid homogeneity and pursue heterogeneity.
I'm aware of that; I commented because I'm surprised by it.
I'm sure the issue was discussed so I'm not implying that they are idiots. But it does seem odd to me.
https://www.fastcompany.com/28121/they-write-right-stuff
https://history.nasa.gov/computers/Ch4-5.html
https://www.nasa.gov/mission_pages/shuttle/flyout/flyfeature...
When I read about these multiply redundant systems I always want to know what provisions are made for restarting or shutting down a part that is producing wrong answers. Can the Power device reset the machine whose instructions lost the vote? If the two cores disagree, do they both reset?
Also, what is done for procedures that must not be interrupted? E.g., once de-orbit is initiated, if the microcontrollers get no new instructions, do they have enough to re-enter safely? Or do they need updates whenever something is supposed to happen?
Assuming they dumb enough to put chromium on a spaceship for a UI then there a good chance that yes they did use bower/npm. Unless they designed it in pure HTML/JS... again this seems unlikely ... given the blunt stupid of the initial choice.
C++ and linux do not strike me as mission critical piece of software / language either. My heart desires for something with verified properties, advanced languages with integrated proof checkers, verified OSs, or at least smaller purpose built OSs that were carefully reviewed. But I guess SpaceX is not to blame for this. It's likely such approaches are simply not ready yet. And I know that C++ and linux are common in mission critical applications. JSF avionics is written in C++ for example.
(Not to mention JavaScript in space, in a mission critical role. That shouldn't have happened.)
How correlated are your layers of swiss cheese...?
I disagree. It is easy to build something presentable with HTML and browsers, and there have been many JS frameworks that lower the barrier. However, that is not because JS is a good language for it; it is because JS is the only language available in that environment.
In fact, there are at least two other languages that transpile to JS, because you need JS to be able to run in a browser. But JS the language is not good enough, so you want two write your code in something else.
Building a UI in a completely different language, without a browser, is definitely harder, but you might have several advantages in the other, non-UI parts.
- first class functions (callback style is a good fit for UI programming)
- async/await and single threaded - many components of UIs are async by nature, Javascript supports you well here while keeping things simple with its single threaded model
Maybe they couldn't afford the Qt license.
I like C# and Rust too. I've built UI's in C++ and MFC/OpenGL/DirectX. It remains my fairly strong opinion that JavaScript is well suited to UI development.
I have encountered this -- for example parsing very large incoming JSON payloads -- but very rarely. And if you really need to mitigate it, there are webworkers.
Chromium is being used in the UI. Not on critical flight software.
> maybe even manual docking
It isn't done manually
> C++ and linux do not strike me as mission critical piece of software / language either
A lot of flight software is done on C++, as you even mentioned. In the end, your best reliability partner is not using a language that's only know by fewer people.
What would be a good "mission critical piece of software"? QNX (with no memory protection?). Linux is fine. I suppose they're not firing Ubuntu out of the box and running that with no customizations.
Separation of concerns and redundancy usually beat correctness on the long term (which doesn't mean they didn't test the correctness of the system).
http://www.qnx.com/developers/docs/6.5.0/topic/com.qnx.doc.i...
zero chances that it's the case. I meant that in aerospace applications UI is mission critical too.
It is, but usability of the UI is mission critical as well. The humans in the capsule need to be able to quickly and effectively consume information in a way that doesn't make important facts hard to get at, and then quickly and effectively perform commands to address issues.
I don't think a touch GUI is a bad choice in this regard, and if I'm building a touch GUI I'd rather use some existing framework. My personal choice would've been something like Gtk or Qt though :)
You can think as each string being “functional“: given the same set of inputs you expect the same set of outputs.
"Real men don’t use backups, they post their stuff on a public ftp server and let the rest of the world make copies."
Real programmers don't use persistent storage, they just rely on physical reality in general.
- don't do dynamic any memory allocation, has bounded memory usage, never runs out of memory.
- has bounded execution times
- has low interrupt latency
- can do checksum on data and internal state an option. (if need to load into memory)
- is tested using the most rigorous standards, 100% verification and code coverage.
If you write great relatively general but customizable database system like this with good indexing, you need only 5-10 customers and you are set. Safety-critical systems are growing in size and there is real need for software that can get easily certified.
SQLite could probably get certified as a part a system at least as read only database. It's relatively solid code. But it will cost big money and I don't think anyone has done it.
I don't have the data to make that call, but I would think someone at their end evaluated this quite thoroughly and determined the risks of rolling their own to exceed the risks of using Chromium.
(Also they mentioned on the stream that they do in fact have a few hardware buttons for important things. Can't cite this, sorry...)
These languages have the best practical static code analyzer, verification and proofing tools money can by. I'm personally using Astree https://www.absint.com/astree/index.htm Automatic docking software for the ATV that delivers supply to ISS is written using C code and verified with Astree.
People get stuck into the language, but if it has some features that make it work, you can use it with verifiers. Then you write tests and simulators and tests ...
JavaScript and Chromium seems more suspect to me, but if the code is well tested and has limited run time, I think it can be good to go. The state is stored and prosessed in those radiation hardened RAD750 processors. Chromium/JavaScript runtime image can restart every few seconds if needed.
The UI and controls seem like unnecessarily flashy bullshit to me. Usually if you diverge from KISS principle bad things happen. But before I make any judgements I would see how they made those decisions.
This feature of Unix and Linux allows controlling the hardware with any programming language capable of reading and writing files, including Python and standard C++. I guess that they may be using standard PC hardware with industrial IO card. They may also use a single-board computer or custom board with x86 low-power variants processors built for embedded applications as SOC system-on-chip. One example of low power x86-SOC based processor is: https://www.cnx-software.com/2015/04/09/vortex86dx3-is-a-new...
I wonder if they had to change any of this for the human rated mission? Would be super exciting to hear tech talks from Spacex!
https://mobile.twitter.com/NASA/status/1266885097359388672
That'd be a really nice touch, one of my favourite fonts for effects. Released after client who commissioned it, a chain of retail bookstores, rejected the design. Largest deployment till now, although just dumped in redesign, was for headings at classifieds website OLX. Caught eyes for a while I guess.
No need then for an especially legible font for cockpit screens like Airbus required https://b612-font.com/ following decades of practices in avionics. Of course the use-case is different and I'm sure everything is adequate for SpaceX goals. Like with the several layers of spontaneously rebooting virtual machines in Tesla cars, in the end branding looks gorgeous.
Also, only the actual graphical display application uses Chromium/JS. The rest of the system is all C++. The display code has 100% test coverage, down to validation of graphical output (for example if you have a progress bar and you set it to X% the tests verify that it is actually drawn correctly).
As far as independent, completely independent from each other.
Indeed you can, judging by the instruction to power off and clean all displays (then power them back on) prior to ISS rendezvous/docking in the live stream.
This is how you get the Cylons pwning your spaceships. >sigh<
But they were experts in using it, some even say Lords
My wonder then is how to effectively manage when things go wrong. The Apollo spacecraft were able to be essentially hot patched directly in the cockpit via the dsky. Can you do that with a fancy touch screen interface now? Perhaps up/down links to the capsule from earth are good/reliable enough to send patches from ground control, if need be? I wonder how the astronauts feel about a potential “loss” of control in case something goes wrong?
On a more serious note. Some of the amazing stuff that the SpaceX team are accomplishing really puts into perspective the day to day work I and others here do.
https://www.theolognion.com/unreal-engine-5-is-meant-to-ridi...
That's not just hilarious, that's also pretty close to how I feel most of the time.
I mean, some people have built the Airbus A380.
On the other hand, last week, I created a test DB with the user "test" and the password "test". I couldn't connect to it, and after a long debugging session, noticed I misspelled "test".
So, yeah...
https://www.latimes.com/archives/la-xpm-1999-oct-01-mn-17288...
So, don’t feel too bad...
Is there a book that lists many bugs and weird IT stories? Like the 500km email or the FileNotFound boolean.
https://en.wikipedia.org/wiki/List_of_software_bugs#Space
I'm sure there's plenty more, albeit of lesser significance.
leftpad managed to expand from planet earth to the ISS. well played!
#universalpaperclips
That doesnt means its perfect. The two 2003 Mars rovers experienced a bug in the then-newfangled flash memory driver. It was successfully fixed and reloaded from Earth.
https://en.wikipedia.org/wiki/VxWorks
https://blogs.windriver.com/wind_river_blog/2019/09/vxworks-...
This damn snake can be compiled on anything.
I'm curious about this system, but not curious enough to spend a week playing with it.
I think it is Wind River Systems.