ECUs have been notoriously badly programmed for years, they're usually contracted out rather than built in-house.
Across wide swaths of Fortune 500 companies, they abhor the idea of having tech people on staff (for reasons I will not go into here) and just now it's starting to bite them in the ass hard.
Proper discipline and established standards are, well, not established, so we get companies serving up product experiences to hundreds of millions of individuals where they are held together by duct tape and super glue (with varying degrees of quality but generally leaning towards overengineered+badly implemented).
That said, there are some standards in place for these devices, but regulation is lax and punitive action rarely taken (it's an honor system).
Compensatory and reactive costs are going to continue to rise as the steady flow of bad code borne from incompetence continues to accrue at exponential rates as more and more companies need tech built but aren't willing to expend the resources for it to be built properly.
Thankfully, as mentioned in the article, some auto companies are starting to embrace the idea of maintaining a healthy staff of developers in-house. Tesla, notably, has over-the-air updates pushed to vehicles directly rather than requiring owners to bring their cars to the dealer.
Having worked in exactly the same domain , I can very well say that OTA is risky, moreover the data providers for the car is not really happy about OTA ( Sprint , AT& T provides connectivity for the projects I worked on.). They lease out a certain bandwidth to the connected cars which sends small bursts of data at a low contracted price. If the car manufacturers need OTA, they will have to pay a premium which will be slapped on the customer. There is also a technicality of downloading a huge image ( possibly 1 GB ) over a spotty network. ( a running car will switch cell networks , network types 3G , LTE , 1x , lost connections etc ).
Why only Tesla has OTA? Some people really think nobody in the industry ever thought of that since the internet was born? Musk is doing a nice publicity stunt.
Tesla sold, what?.... 30000 cars all time? All models above $50000? So addressing people living in places where OTA is not an issue.
How are all the other car makers going to handle a production of a few milion units PER YEAR, being updated OTA? How are they going to update cars sold in countries where Internet is a dream?
If you change your release strategy to fit an OTA update model, you better be able to support it. This is not a problem for the volume Tesla is selling, and neither for it's audience.
That doesn't make sense to me because car companies hire huge numbers of automotive engineers (I think?)
I always thought this was an engineering culture war thing -- software engineers aren't really engineers and software is sort of this silly thing that you use to glue together "real" engineering products.
Hopefully the author of your parent post will elaborate some :-)
(edit: I assumed you meant business-engineering culture war, not engineering-software. Might've been mistaken upon rereading)
The culture war is alive and well, many companies (mine included, unfortunately) treat software engineers as barely qualified typists of the "algorithms" designed by "real engineers". They'd get rid all of them in a second if they could, every so often a new manager tries.
When I was growing up in the 80's, my father was a software engineer at Honeywell and said they discriminated against the software guys, even when they knew software was the future of computing. He said it had been going on since the 60's and 70's as related to him by previous employees. He was told if he was a software engineer to just keep his head down and not talk to the "real engineers" working on "real problems".
Likewise when I was in college for computer science, he reminded me of the same struggles he had when I used to get fed shit from the other engineering students about how "If all else failed, they could become a CS major." and just be nice because one day they'll be coming to me for help.
And here we are today. Same thing, same arguments, same outdated viewpoints.
Well, unfortunately many software engineers don't behave like real engineers, so is it any surprise?
I mean to say, most people here will be highly familiar with version control, but back in the days when I used to audit companies, version control was barely[1] used, and editing files on the live server was the norm. And even here we're still having discussions about how "hard" it is to manage dependencies, use strong typing, use safe languages, garbage collection, bounds checking, formal proofs, etc.
[1] I think I may be correct in saying never used. I can't recall a case where we saw a company using any kind of VCS, but perhaps I just forgot.
Your criticism is valid from our perspective, but the non-software engineers on the other side of the culture divide are even less aware of proper software practice, so I don't see how the failure of software people to follow those practices could be the source of the prejudice.
I think it's a different problem. Software engineering is fundamentally about the management of complexity. People who have only worked on small programs -- like most non-software engineers and scientists -- don't understand the need to manage complexity because they've never worked on any program large enough to require it. Programming in the small is easy, particularly when you're mostly doing numerics. Other engineers see that and think that all programming must be easy.
Whenever I describe software engineering I typically use some form of "a true software engineer doesn't just make something work, he avoids &#%$ing over the next person to pick up his or her code."
Funny that you include garbage collection as part of what we should be doing, in an article about software in cars. Garbage collection is almost never the right answer for software in embedded systems.
Unless the poster was including ARC-style systems in what they meant by "garbage collection".
Ideally you want something that is both deterministic (i.e., it's not running randomly in the background like a traditional GC) and that guarantees there are no leaks. In theory ARC gives you both of these.
ARC depends on malloc/free which isn't quite as deterministic as you make it out to be - precisely because malloc/free does not (typically) run in the background, it needs to update internal datastructures when it gets the chance, which can lead to considerable latency spikes.
In fact, I'd go so far as to say latency isn't the issue - the issue is that typically GC'd languages rely heavily on heap allocation because it's cheaper(!) not more expensive. Put another way: java creates lots of garbage, C/C++ does not.
If you wrote a C/C++ program that allocated as heavily as a typical java program... well, I'd expect performance issues that would probably be worse than java's.
True, any malloc (or new) and any free (or delete) is non-deterministic (unless you're using a placement allocator). But GC isn't better, because the new still has the same non-determinism, Also, the GC has to do more work than a programmer-controlled delete (walking data structures for liveness, or something approximately equally expensive), and tends to do both the walking and the deletes in bursts. But...
> java creates lots of garbage, C/C++ does not.
Properly written C/C++ does not. Badly written C/C++ creates garbage (leaks memory), and it can never be recovered, because there's no GC. The solution isn't GC, though - it's to pay more attention to who owns the memory.
Yes you are absolutely right. The connected car project which I worked for had ECUs outsourced. The car manufacturer said "hey we are a car company". You deal with the technicalities of ECUs giving you proper data.
The ECUs were giving me frames with bits turned on at the wrong places. We used to call the telematics hardware/software provider which gives us an abstracted API to talk to ECUs quoting the problem. Their reply was "thats the ECUs messing up, we will "fix it" by giving you the bit turned on at the correct position or you change your code, because changing the API needs an OTA and OTA is not supported for that car manufacturer.
Which I think is symptomatic of the bigger issue. Car companies look at anything that goes into a car as a physical part, and they've gotten extraordinarily efficient (through ruthless competition) at process for dealing with physical parts.
Except... anything that contains code has orders of magnitude more test dimensions than a typical physical part.
I would probably stop driving altogether if I knew exact figures on how many auto manufacturers even have in-house test coverage of their outsourced ECU / microcontrollers.
What makes you believe in-house SW is more reliable? Anecdotal or not, I know people who worked for both a X's (German car maker, not VW) supplier and X themselves in SW engineering and he told me the supplier was on a totally another level above the in-house development process.
Suppliers also have tradition in car sub-systems. They also provide complete sub-systems (hardware and software). Much easier to implement the SW when you also build the full HW (injectors, pump, rail, sensors, ECU, etc.). You should see some of the development labs. Tens/hundreds of millions of EURO invested in equipment over time.
The only guys I know, who are still keeping a lot in-house, are the Japanese car makers (though they work with some locals in Japan, they can't be that independent). That market is opening a lot more to EU suppliers in next few years for sure.
In general car makers just lack the technological know-how to do a lot of stuff in-house. If they give up suppliers tomorrow, gonna take many years just to catch-up.
The old car saying goes: Fast, Cheap, Reliable. Pick two, you can't get the third.
Requirement creep seems to be the real problem here: Now we want aggressive fuel-efficiency targets, crash safety, and a host of other "nice to have" modern features. And it's in the computing unit which ends up trying to mediate these necessarily competing demands.
From the article: "New high-end cars are among the most sophisticated machines on the planet, containing 100 million or more lines of code."
Does anyone know where this number comes from and if its realistic? It's hard for me to imagine unless you're counting all the LOC from Microsoft Windows: Car Edition or something.
Can't answer where they came up with that number but it does seem rather larger. The only comparable machine where we do know the number is the Mars Curiosity Rover running somewhere around 2.5 Million LoC. Now, given that there are upwards of 200 micro controllers in a luxury car, I can see that number growing exponentially, but 100 Million???
2.5 MLoC sounds extremely high even for a Mars Rover. I really wonder how it's counted? Is it C code or ASM instructions? Does it include all standard libraries that get referenced, even if they end up not being compiled into the final binary? Or is everything just written in Java and configured with XML?
Yes, that has always been my assumption -- they're probably counting operating system, libraries, etc. in those mlocs. It's X MLOCs in the car, not X MLOCs written by the car manufacturer.
We're not just talking about one computer here -- it's not unusual for a modern car to have ~70 ECUs. Of course, many of those will be quite small, simple devices.
"One option for making auto software safer is to open it to public scrutiny. While this might sound counterintuitive, some experts say that if automakers were forced to open up their source code, many interested people — including coding experts and academics — could search for bugs and vulnerabilities. "
This is simply not going to happen.
Firstly, the majority of automotive control systems on an ECU are running generated code from toolschains such as CATIA Systems, Dymola, MapleSim, OpenModelica, SCICOS, SimulationX, Vertex etc etc etc.
To validate against vulnerabilities requires in-depth knowledge of a particular manufacturer's modeling practices & standards. The code is not readable, the model is.
Which relates to the second reason, the IP in these models is very closely guarded. Competitive advantage and product differentiations are pretty much all software now. Even suppliers and manufacturers share black box models & systems back and forth for fear of leaking IP.
In a world where say BMW hands Bosch a set of requirement, Bosch supplies BMW HW and BMW validates in simulation and analysis with neither getting insight in to the other's design. You're telling that needs to now be done in the open?
Well, what you're saying is that they wouldn't open source it voluntarily. This is why we have regulations: to make people and companies do things that they wouldn't do voluntarily.
Yes it is. And if hundreds or thousands of people had died from windows vulnerabilities and tremendous environmental damage had been done, that might well have happened.
Most of the security breaches are due to people doing stuff really wrong not just a late patch. And even then I haven't heard of anyone dying as a result yet.
People have probably died (if you believe public health stats) as a result of increased NOx production of these vehicles. There's also been a lot more pollution. Big, big pollution problems have caused substantial changes to things, like the creation of the EPA here in the US.
Just like people stop mining for coal or iron when it causes loss of life and environmental damage?
The people being harmed by software vulnerabilities are rarely the people who made an informed decision to use a certain piece of software. People who shop at Target can't meaningfully be said to make a choice to trust Target's software -- it's not something the average consumer things about.
So although "should" is an understandable prescription, "would" is a bit unrealistic...
If there were any alternatives to coal or iron that wouldn't cause as large loss of life, then we would have switched to those. If you find out your car is unsafe to use, you switch to a different one, because other cars offer similar functionality - the same with Windows, if it was so full of bugs that it was an actual danger to use, then people would simply switch to Linux,*BSD or MacOS, it's not like there are no alternatives.
The behavior you're describing is disproved by history. People use demonstrably insecure products that are insecure all the time, even when great secure alternatives exist.
Even speaking theoretically, the behavior you're describing isn't even an accurate description of how rational actors without resource constraints and with perfect knowledge would behave (because e.g. there are qualities aside from security and safety that might make it impossible to use an alternative product, and there might be qualities that are impossible to measure directly). And anyways, all three of those assumptions are clearly out-of-touch with reality.
Specifically wrt automobile software:
* Even with public access to source code, it's impossibly expensive for the average consumer (even the average software engineer; hell, even the average automotive engineer) to make a decision with truly perfect knowledge. Any number of safety constraints boil down to intractable questions if code is designed in a way that isn't amenable to inspection. So even in a purely theoretical setting with completely rational actors, making a fully informed decision might just not be possible.
* Many consumers will overlook errors in previous products and plausible errors in current products; Toyota's brand is doing just fine despite the findings of experts in the UA case. And I bet a lot of the software on cars that have sold even this year has some of the same structural problems, because based upon the expert testimony, a lot of that code probably needed to be essentially rewritten, and it's highly unlikely that they've achieved that in a year or two.
The difference, to play da, is that both of your points get solved with greater transparency.
Is Linux more secure because everyone (quiet, Gentoo crowd) audits their own source? No, it's more secure because:
Anyone can choose to audit the source with minimal roadblocks.
Anyone who thinks they've found an issue can trumpet it from the rooftops.
Together, this means that even though I might not line-by-line audit my system, I can (with varying levels of paranoia / assurance) obtain a system that I have confidence others have audited. And furthermore, if there are issues found in the future then I can be reasonably sure of hearing about them.
Neither of these benefits is possible with closed source embedded systems.
(Or, from article "Ninety-nine percent of the buyers would never read anything. But out of the 11 million people whose car was cheating, one of them would have found it,” he said. “And Volkswagen would have been caught in 2009, not 2015.”)
> both of your points get solved with greater transparency
My first point was exactly that transparency isn't a solution, even theoretically. There's no such thing as perfect knowledge when the questions you're asking are undecidable.
My second point was that even if transparency is a solution theoretically, it's demonstrably not a solution historically. People (including customers not subject to CYA-induced stupidity) choose insecure proprietary systems over secure open source systems all the time.
This raises a point that has confused me for a long time. Aerospace engineering is known for its track record of safety (not to say incidents don't happen, but they're rare, and usually self-contained). Why, then, is the same level of rigor not applied to the automotive industry, by government (or even international) mandate? Just as much damange can just as easily be done, and if it was possible to seize control of a Boeing 747 from your living room, you can bet there would be people fired within the hour, and arrested within the day.
That said, something tells me that, in light of recent events, there will be a push for the automotive industry to start using things like DO-178, and its companions for mechanical and electronic engineering (whose numbers I don't recall).
Air travel is inherently more dangerous than automobile travel. This is intuitively and actually true. People learn from a very young age that falling hurts and falling even 10 meters can be fatal. Falling, flying, great heights tickle the fear centers in your lizard brain. Vehicles are more like incremental improvements to things that are safe like running or riding a horse. Additionally auto accidents are mostly isolated to 2 parties on the roadway they're travelling. So we don't exercise the same caution with cars.
Practically, air travel is safer because they are built, maintained, piloted, and continuously monitored by professionals. Conversely a 15 year old can build a kit car and drive it on the highway legally. If they break a dozen traffic laws they may or may not get caught and penalized.
If we could get people to assess and respond to risk rationally, we could make a lot of things safer.
The psychology of it is interesting, isn't it? Flying in a plane seems so intuitively unsafe that we go to great efforts to make it safe, while traveling on the ground seems so intuitively safe that we don't worry about it as much. (Remember how hard it was to get most people to wear seat belts?) And yet, far more people die in car crashes than plane crashes.
While Microsoft's product issues do impose externalities on net users at large, that pales in comparison to letting 2 ton machines travel 60 MPH on the roads, while running closed source software.
Simply require source code publication for cars that are to be made street legal.
It's not going to happen unless legislated. That's the whole point. Currently there is no legislation forcing it to happen, but in the future there might be. That might be a good idea, even if the various companies involved don't want to do it.
There are a lot of bad reasons for "closely guarded" IP:
1. The government told you to put spyware in e.g. your mobile baseband processor
2. Your GPU violates a lot of patents
3. You ECU implements dark patterns w.r.t. emissions testing
Make regulations requiring publishing the code, and require making the toolchain available to code can be verified vs. the shipped image. Proprietary software isn't a valid excuse when the stakes are to poison the environment.
I agree with you -- the push for opening up code to public scrutiny is politically and economically unfeasible. But that doesn't mean nothing can be done -- a more palatable regulatory regime might require oversight a la the aerospace industry, but emphasizing the use of formal methods over expensive and arbitrary process hoop-jumping.
No problem for that.
Are you ready to pay to have that? Because stuff like that does not come cheap. Quite sure we can find some numbers on the Internet, probably you are not the first people in the world to think of implementing aerospace regulation in the automotive business :-). I am pretty sure in an industry where saving 12$ / unit (trough some improvement for example) is considered a GIGANTIC gain, applying some super strict aerospace standards will take the car prices to unacceptable levels.
Okay, then car companies can open source their ECU (and other safety-critical) software.
No one is recommending copy/pasting aerospace regulations. In part because we're now 10-30 years further along in basic and applied CS research than we were when those regs were drafted.
What makes aerospace regulations expensive is the process, not the analysis. The attitude that formal methods are completely untenable is painfully sutck in the 1990's. For god's sake, Facebook runs portions of their code base through static analyses, as does Google. If websites are important enough to establish the safety of using formal methods, then the software running are ours sure as hell is.
In any case, thing is sure: as software becomes an integral part of high-level safety-critical control decisions, it needs to either be taken seriously by car companies and trusted by the general public.
Formal methods and static analysis are the only way we know of for establishing trust in software systems without revealing their source code or relying on black box testing (the downsides of which are demonstrated by both the Toyota UA and VW emissions testing cases). If you know of a third option, then by all means...
True, maybe a little bit more openness would not hurt nobody. Probably would help people trust automakers more and not go nuts when a scandal goes live like the VW.
Never seen any banks open their strategy to the public after so many of them fucked us over a few years ago. It's not like I can copy their strategy and go make me another bank.
But going back to automotive, because that's what I do for a living. The process is quite strict and controlled. Efforts is done and periodic audit is done too to ensure adherence to ASPICE (http://vda-qmc.de/en/software-processes/automotive-spice/) and other regulations, including customer audits, which are tough. We do have internal processes, coding (and not only) methods, source control, configuration management (features, bug tracking) across all project and generic development base.
The basic workflow is follows a V-cycle path:
- receive requirements.
- requirements engineering, experts involved;
- test design based on requirements;
- internal design of whatever kind (models etc.);
- design reviews, simulations (if models), all kind of analysis;
- implementation (if not code generation);
- code reviews;
- static analysis ((pc-)lint, klocwork to name a few);
- module tests (unitary tests (ptu), tests on the HW with simulated signals - aka tetsbench tests);
- integration tests;
- system validation tests (requirements tests), hardware-in-the-loop tests, close loop tests;
- dependability tests run on the car.
In parallel the system is calibrated and calibrations are used during validations. For every version going into production (because some are just interim steps for validation purpose), there's a lot of testing going on on features already tested before.
SW execution is monitored by SW. The micro is monitored in some interesting ways and on top of that an IC is monitoring all the... other monitors. Lots of stuff is done to ensure RAM/ROM consistency, process execution monitoring, signal acquisition plausibility. Many inputs/executions diagnostics are requested by the laws, but a lot more are done for system robustness.
Not gonna talk about HW design and validations. They have their own set of regulations and standard tests to fulfill, which I don't know that good.
This is where the car maker takes over from the supplier and does their own set of tests (which I don't know in much detail):
- tens/hundreds of thousands of miles on a fleet of vehicles, recording all that happens and future analysis follows
- cold trip, hot trip plus various trips to test response to different altitudes, sea side effects etc.
- lots of garage testing
- regulations tests (the car must respect various regulations of the countries it targets... OBD regulations, CARB, EU stuff, etc.)
A lot of other tests are carried out in different phases of production. For example a special SW which runs on the production line is designed to test all circuitry one last time before flashing the real working SW.
Too many people have the impression this SW is build "on our knees" (an expression that sounds good in my native language), but this is far beyond that. The threat of car recalls (which is, in the end, charged to the supplier causing it, or the car maker if it is his fault) and fucking up all the profit is not stuff engineering plays with. There are over 1k engineers working on engine ECU R&D in various locations (from system design to SW and tests).
Obviously open sourcing the software is the only solution and everyone here knows that. Software security isn't a new problem, it's a many decades old problem with simple, effective procedures that can lead to proper solutions. Implementing those procedures and solutions isn't enough, as the most important part of security is trust and there can be no trust without transparency. Period.
Software should be required to be open source if the authorities that be can't police it (and they can't). Let the people do for themselves what the government can't. I can't think of a single, logical reason why it shouldn't be open source (profit is not a logical reason when lives are at stake). Until more people die, of course there will be no significant changes in this industry. In fact, according to history, until enough people die to reach the threshold for anger in the public at large, there will be no changes in this industry.
Opensourcing is not a replacement for QA practices.
If it was, open source applications would never have critical bugs or security vulnerabilities.
It may mitigate some issues in a long run (e.g. intentional backdors) but in a short run it will create havoc as access to source code will make it easier to create exploits.
And suddenly you have whole new problem of making sure that everyone keeps their ECU software up to date or they risk fatal crash.
"Opensourcing is not a replacement for QA practices."
I'm not suggesting it is, simply that public oversight (open source) is the only way to ensure trust of a secure system. The system itself still has to be secure and requires QA like any other.
The only people who will understand that code (assuming is not just generated code) will be people from competitors... maybe. People outside the business will have no clue. It will be a continuous source of false alarms. Will be really hard to filter out the crap. Other people already talked about exploits. Every day you'll live the fear of "do I run up to date SW on my car???". This is not a phone.
First you need to know the system, then you can understand what the code should do then you can have a fare chance of doing a code inspection.
I think people parallelize too much to web development or other environments a lot more exposed to public development.
People think that if you can read someone else's C code for voice communication, they will also be able to understand the C code behind a diesel injection system, air intake path, exhaust gas treatment, fuel mass setpoints calculation. If you have never been exposed to stuff like that, you cannot fully judge if the code is wrong. You may find something obvious here and there (like a loop that may go beyond the limit in theory but never in the field).
And believe it or not, there is still competition for IP. Technical strategies of doing something in certain ways giving supplier X clear advantage in some field. Why would X share his knowledge just like that? We are talking about huge money. Never seen Google share the source of all they do.
53 comments
[ 3.0 ms ] story [ 42.3 ms ] threadAcross wide swaths of Fortune 500 companies, they abhor the idea of having tech people on staff (for reasons I will not go into here) and just now it's starting to bite them in the ass hard.
Proper discipline and established standards are, well, not established, so we get companies serving up product experiences to hundreds of millions of individuals where they are held together by duct tape and super glue (with varying degrees of quality but generally leaning towards overengineered+badly implemented).
That said, there are some standards in place for these devices, but regulation is lax and punitive action rarely taken (it's an honor system).
Compensatory and reactive costs are going to continue to rise as the steady flow of bad code borne from incompetence continues to accrue at exponential rates as more and more companies need tech built but aren't willing to expend the resources for it to be built properly.
Thankfully, as mentioned in the article, some auto companies are starting to embrace the idea of maintaining a healthy staff of developers in-house. Tesla, notably, has over-the-air updates pushed to vehicles directly rather than requiring owners to bring their cars to the dealer.
Tesla sold, what?.... 30000 cars all time? All models above $50000? So addressing people living in places where OTA is not an issue.
How are all the other car makers going to handle a production of a few milion units PER YEAR, being updated OTA? How are they going to update cars sold in countries where Internet is a dream? If you change your release strategy to fit an OTA update model, you better be able to support it. This is not a problem for the volume Tesla is selling, and neither for it's audience.
Ah, go on. I'm guessing it's partly a Two Cultures problem (not wanting to give tech people status) and partly a cost problem?
I always thought this was an engineering culture war thing -- software engineers aren't really engineers and software is sort of this silly thing that you use to glue together "real" engineering products.
Hopefully the author of your parent post will elaborate some :-)
(edit: I assumed you meant business-engineering culture war, not engineering-software. Might've been mistaken upon rereading)
When I was growing up in the 80's, my father was a software engineer at Honeywell and said they discriminated against the software guys, even when they knew software was the future of computing. He said it had been going on since the 60's and 70's as related to him by previous employees. He was told if he was a software engineer to just keep his head down and not talk to the "real engineers" working on "real problems".
Likewise when I was in college for computer science, he reminded me of the same struggles he had when I used to get fed shit from the other engineering students about how "If all else failed, they could become a CS major." and just be nice because one day they'll be coming to me for help.
And here we are today. Same thing, same arguments, same outdated viewpoints.
I mean to say, most people here will be highly familiar with version control, but back in the days when I used to audit companies, version control was barely[1] used, and editing files on the live server was the norm. And even here we're still having discussions about how "hard" it is to manage dependencies, use strong typing, use safe languages, garbage collection, bounds checking, formal proofs, etc.
[1] I think I may be correct in saying never used. I can't recall a case where we saw a company using any kind of VCS, but perhaps I just forgot.
I think it's a different problem. Software engineering is fundamentally about the management of complexity. People who have only worked on small programs -- like most non-software engineers and scientists -- don't understand the need to manage complexity because they've never worked on any program large enough to require it. Programming in the small is easy, particularly when you're mostly doing numerics. Other engineers see that and think that all programming must be easy.
Ideally you want something that is both deterministic (i.e., it's not running randomly in the background like a traditional GC) and that guarantees there are no leaks. In theory ARC gives you both of these.
In fact, I'd go so far as to say latency isn't the issue - the issue is that typically GC'd languages rely heavily on heap allocation because it's cheaper(!) not more expensive. Put another way: java creates lots of garbage, C/C++ does not.
If you wrote a C/C++ program that allocated as heavily as a typical java program... well, I'd expect performance issues that would probably be worse than java's.
> java creates lots of garbage, C/C++ does not.
Properly written C/C++ does not. Badly written C/C++ creates garbage (leaks memory), and it can never be recovered, because there's no GC. The solution isn't GC, though - it's to pay more attention to who owns the memory.
The ECUs were giving me frames with bits turned on at the wrong places. We used to call the telematics hardware/software provider which gives us an abstracted API to talk to ECUs quoting the problem. Their reply was "thats the ECUs messing up, we will "fix it" by giving you the bit turned on at the correct position or you change your code, because changing the API needs an OTA and OTA is not supported for that car manufacturer.
Except... anything that contains code has orders of magnitude more test dimensions than a typical physical part.
I would probably stop driving altogether if I knew exact figures on how many auto manufacturers even have in-house test coverage of their outsourced ECU / microcontrollers.
Suppliers also have tradition in car sub-systems. They also provide complete sub-systems (hardware and software). Much easier to implement the SW when you also build the full HW (injectors, pump, rail, sensors, ECU, etc.). You should see some of the development labs. Tens/hundreds of millions of EURO invested in equipment over time.
The only guys I know, who are still keeping a lot in-house, are the Japanese car makers (though they work with some locals in Japan, they can't be that independent). That market is opening a lot more to EU suppliers in next few years for sure.
In general car makers just lack the technological know-how to do a lot of stuff in-house. If they give up suppliers tomorrow, gonna take many years just to catch-up.
Requirement creep seems to be the real problem here: Now we want aggressive fuel-efficiency targets, crash safety, and a host of other "nice to have" modern features. And it's in the computing unit which ends up trying to mediate these necessarily competing demands.
Does anyone know where this number comes from and if its realistic? It's hard for me to imagine unless you're counting all the LOC from Microsoft Windows: Car Edition or something.
Highly doubtful, Opportunity is at 4,000+ days of uptime...
This is simply not going to happen.
Firstly, the majority of automotive control systems on an ECU are running generated code from toolschains such as CATIA Systems, Dymola, MapleSim, OpenModelica, SCICOS, SimulationX, Vertex etc etc etc.
To validate against vulnerabilities requires in-depth knowledge of a particular manufacturer's modeling practices & standards. The code is not readable, the model is.
Which relates to the second reason, the IP in these models is very closely guarded. Competitive advantage and product differentiations are pretty much all software now. Even suppliers and manufacturers share black box models & systems back and forth for fear of leaking IP.
In a world where say BMW hands Bosch a set of requirement, Bosch supplies BMW HW and BMW validates in simulation and analysis with neither getting insight in to the other's design. You're telling that needs to now be done in the open?
I do not disagree with you. These are safety critical systems, I just don't see how companies could be forced to share IP.
What is more likely in my view is something similar to the validation standards in aviation: https://en.wikipedia.org/wiki/DO-178B
Most of the security breaches are due to people doing stuff really wrong not just a late patch. And even then I haven't heard of anyone dying as a result yet.
People have probably died (if you believe public health stats) as a result of increased NOx production of these vehicles. There's also been a lot more pollution. Big, big pollution problems have caused substantial changes to things, like the creation of the EPA here in the US.
Then people would stop using Windows, simple as that.
The people being harmed by software vulnerabilities are rarely the people who made an informed decision to use a certain piece of software. People who shop at Target can't meaningfully be said to make a choice to trust Target's software -- it's not something the average consumer things about.
So although "should" is an understandable prescription, "would" is a bit unrealistic...
Even speaking theoretically, the behavior you're describing isn't even an accurate description of how rational actors without resource constraints and with perfect knowledge would behave (because e.g. there are qualities aside from security and safety that might make it impossible to use an alternative product, and there might be qualities that are impossible to measure directly). And anyways, all three of those assumptions are clearly out-of-touch with reality.
Specifically wrt automobile software:
* Even with public access to source code, it's impossibly expensive for the average consumer (even the average software engineer; hell, even the average automotive engineer) to make a decision with truly perfect knowledge. Any number of safety constraints boil down to intractable questions if code is designed in a way that isn't amenable to inspection. So even in a purely theoretical setting with completely rational actors, making a fully informed decision might just not be possible.
* Many consumers will overlook errors in previous products and plausible errors in current products; Toyota's brand is doing just fine despite the findings of experts in the UA case. And I bet a lot of the software on cars that have sold even this year has some of the same structural problems, because based upon the expert testimony, a lot of that code probably needed to be essentially rewritten, and it's highly unlikely that they've achieved that in a year or two.
Is Linux more secure because everyone (quiet, Gentoo crowd) audits their own source? No, it's more secure because:
Anyone can choose to audit the source with minimal roadblocks.
Anyone who thinks they've found an issue can trumpet it from the rooftops.
Together, this means that even though I might not line-by-line audit my system, I can (with varying levels of paranoia / assurance) obtain a system that I have confidence others have audited. And furthermore, if there are issues found in the future then I can be reasonably sure of hearing about them.
Neither of these benefits is possible with closed source embedded systems.
(Or, from article "Ninety-nine percent of the buyers would never read anything. But out of the 11 million people whose car was cheating, one of them would have found it,” he said. “And Volkswagen would have been caught in 2009, not 2015.”)
My first point was exactly that transparency isn't a solution, even theoretically. There's no such thing as perfect knowledge when the questions you're asking are undecidable.
My second point was that even if transparency is a solution theoretically, it's demonstrably not a solution historically. People (including customers not subject to CYA-induced stupidity) choose insecure proprietary systems over secure open source systems all the time.
That said, something tells me that, in light of recent events, there will be a push for the automotive industry to start using things like DO-178, and its companions for mechanical and electronic engineering (whose numbers I don't recall).
Practically, air travel is safer because they are built, maintained, piloted, and continuously monitored by professionals. Conversely a 15 year old can build a kit car and drive it on the highway legally. If they break a dozen traffic laws they may or may not get caught and penalized.
If we could get people to assess and respond to risk rationally, we could make a lot of things safer.
Simply require source code publication for cars that are to be made street legal.
It's not going to happen unless legislated. That's the whole point. Currently there is no legislation forcing it to happen, but in the future there might be. That might be a good idea, even if the various companies involved don't want to do it.
1. The government told you to put spyware in e.g. your mobile baseband processor
2. Your GPU violates a lot of patents
3. You ECU implements dark patterns w.r.t. emissions testing
Make regulations requiring publishing the code, and require making the toolchain available to code can be verified vs. the shipped image. Proprietary software isn't a valid excuse when the stakes are to poison the environment.
No one is recommending copy/pasting aerospace regulations. In part because we're now 10-30 years further along in basic and applied CS research than we were when those regs were drafted.
What makes aerospace regulations expensive is the process, not the analysis. The attitude that formal methods are completely untenable is painfully sutck in the 1990's. For god's sake, Facebook runs portions of their code base through static analyses, as does Google. If websites are important enough to establish the safety of using formal methods, then the software running are ours sure as hell is.
In any case, thing is sure: as software becomes an integral part of high-level safety-critical control decisions, it needs to either be taken seriously by car companies and trusted by the general public.
Formal methods and static analysis are the only way we know of for establishing trust in software systems without revealing their source code or relying on black box testing (the downsides of which are demonstrated by both the Toyota UA and VW emissions testing cases). If you know of a third option, then by all means...
But going back to automotive, because that's what I do for a living. The process is quite strict and controlled. Efforts is done and periodic audit is done too to ensure adherence to ASPICE (http://vda-qmc.de/en/software-processes/automotive-spice/) and other regulations, including customer audits, which are tough. We do have internal processes, coding (and not only) methods, source control, configuration management (features, bug tracking) across all project and generic development base.
The basic workflow is follows a V-cycle path:
- receive requirements.
- requirements engineering, experts involved;
- test design based on requirements;
- internal design of whatever kind (models etc.);
- design reviews, simulations (if models), all kind of analysis;
- implementation (if not code generation);
- code reviews;
- static analysis ((pc-)lint, klocwork to name a few);
- module tests (unitary tests (ptu), tests on the HW with simulated signals - aka tetsbench tests);
- integration tests;
- system validation tests (requirements tests), hardware-in-the-loop tests, close loop tests;
- dependability tests run on the car.
In parallel the system is calibrated and calibrations are used during validations. For every version going into production (because some are just interim steps for validation purpose), there's a lot of testing going on on features already tested before.
SW execution is monitored by SW. The micro is monitored in some interesting ways and on top of that an IC is monitoring all the... other monitors. Lots of stuff is done to ensure RAM/ROM consistency, process execution monitoring, signal acquisition plausibility. Many inputs/executions diagnostics are requested by the laws, but a lot more are done for system robustness. Not gonna talk about HW design and validations. They have their own set of regulations and standard tests to fulfill, which I don't know that good.
This is where the car maker takes over from the supplier and does their own set of tests (which I don't know in much detail): - tens/hundreds of thousands of miles on a fleet of vehicles, recording all that happens and future analysis follows - cold trip, hot trip plus various trips to test response to different altitudes, sea side effects etc. - lots of garage testing - regulations tests (the car must respect various regulations of the countries it targets... OBD regulations, CARB, EU stuff, etc.)
A lot of other tests are carried out in different phases of production. For example a special SW which runs on the production line is designed to test all circuitry one last time before flashing the real working SW.
Too many people have the impression this SW is build "on our knees" (an expression that sounds good in my native language), but this is far beyond that. The threat of car recalls (which is, in the end, charged to the supplier causing it, or the car maker if it is his fault) and fucking up all the profit is not stuff engineering plays with. There are over 1k engineers working on engine ECU R&D in various locations (from system design to SW and tests).
Software should be required to be open source if the authorities that be can't police it (and they can't). Let the people do for themselves what the government can't. I can't think of a single, logical reason why it shouldn't be open source (profit is not a logical reason when lives are at stake). Until more people die, of course there will be no significant changes in this industry. In fact, according to history, until enough people die to reach the threshold for anger in the public at large, there will be no changes in this industry.
If it was, open source applications would never have critical bugs or security vulnerabilities.
It may mitigate some issues in a long run (e.g. intentional backdors) but in a short run it will create havoc as access to source code will make it easier to create exploits.
And suddenly you have whole new problem of making sure that everyone keeps their ECU software up to date or they risk fatal crash.
I'm not suggesting it is, simply that public oversight (open source) is the only way to ensure trust of a secure system. The system itself still has to be secure and requires QA like any other.
First you need to know the system, then you can understand what the code should do then you can have a fare chance of doing a code inspection.
I think people parallelize too much to web development or other environments a lot more exposed to public development. People think that if you can read someone else's C code for voice communication, they will also be able to understand the C code behind a diesel injection system, air intake path, exhaust gas treatment, fuel mass setpoints calculation. If you have never been exposed to stuff like that, you cannot fully judge if the code is wrong. You may find something obvious here and there (like a loop that may go beyond the limit in theory but never in the field).
And believe it or not, there is still competition for IP. Technical strategies of doing something in certain ways giving supplier X clear advantage in some field. Why would X share his knowledge just like that? We are talking about huge money. Never seen Google share the source of all they do.