One of the most high-profile cases I can think of is the Tylenol murders and subsequent recall. The J&J handling of the problem is actually a textbook case of how to handle this kind of thing properly.
There's a whole area of business called "crisis management." Some of the more famous case studies are the Tylenol poisoning case, the Ford Pinto controversy and lawsuit, the Valdez oil spill, Deepwater Horizon, the Firestone tire controversy, and it wasn't even that long ago that there was a small case of mass hysteria regarding Toyotas accelerating all on their own that managed to spread to other brands of cars before people realized that these were just careless drivers. As for abandoning a new product, I suppose there's not that many industries where you come out with regular updates like this every couple of years and it's a big deal. I know Microsoft has rebranded a couple of products before. They rolled out Windows 7 instead of trying to convince everyone that Vista was all fixed and not as bad as they say. I suspect this will be a similar case. Hold off until the controversy dies down, then basically just release the current iteration of their phone when people calm down and they can make sure the thing barely runs warm ever.
This is completely not a technical decision, it's a branding and marketing decision.
Even though the percentage is vanishingly small, Samsung cannot be associated with exploding phones.
Even airlines warn against Samsung Note 7s specifically during preflight briefings. They're amputating part of their business to stop contagion to the rest of their brand.
I thought so too. I meant it's just been three explosions, or so I heard. However this article claims 97 incidents with breakdowns on overheating vs burning. It seems pretty significant. I mean, not statically against devices in the wild, but it takes a relatively few amount of safety incidents to question an entire product line.
>I thought so too. I meant it's just been three explosions, or so I heard.
Far more than 3. It's already more than 3 on the "replacement" "fixed" 7s -- it's dozens for the first version, including in planes, while driving, etc.
This "vanishingly small" number is still way, way too high. Five in a week? If there's one thing you don't want your electronics to do it's potentially kill people.
There was a rash of exploding battery cells in the 2006-2007 era where Apple even issued a recall. At the time Apple was still using off-the-shelf cells and hadn't started to make their own batteries. This affected many other vendors using the same suppliers.
Even Boeing's planes were not immune to battery problems, with the 787 famously suffering some serious issues, including fires.
Battery technology is something that's very hard to get absolutely right, to render something safe enough it won't explode. You want a lot of "sigmas" in your quality to avoid any serious defects.
I don't know what Apple does to their batteries, but the sorts of abuse they can apparently endure without catching fire is remarkable: https://www.youtube.com/watch?v=iy5kAY5SqEk
I don't know what Samsung did to cause all these explosions, but I hope these are thoroughly investigated.
If it's a manufacturing problem, that's one thing, but I have a sneaking suspicion this could boil down to software. You can measure your manufacturing process very closely, but testing embedded software, such as in the battery itself, is a total nightmare by comparison.
> Even though the percentage is vanishingly small, Samsung cannot be associated with exploding phones.
It's still high enough to indicate a general manufacturing problem. To ditch it completely may mostly be a marketing decision but to put production on hold for some time to investigate and really fix the root cause seems to be reasonable. It may also help internal processes making sure that no additional item from a possible faulty batch ever gets sold accidentally. Maybe we will see a similar device in a few months, maybe branded differently, maybe with a redesigned casing, maybe some marketing campaign showing how well they've protected the battery this time.
I disagree. Samsung seemed to have proactively started replacing phones. Even before the media hyped it up. When they tried replacing, it still would overheat. Eventually management gave up mostly because as others have said they couldn't figure out why the phones were heating up in the first place.
Can confirm. Just took a flight yesterday and noticed the Galaxy 7 warning had become part of the official spiel read by the gate attendants when announcing flight info.
One of the more comical attendees turned it into a joke, something like '... If you would like to stow your larger carry on, please come to the counter.... If you have a Samsung Galaxy 7, please place it in the garbage prior to entering the aircraft...'.
It was a crowded flight with a long line and already delayed. He was just trying to lighten up the atmosphere, and everyone did laugh, so it was clever.
For Samsung, this is a marketing nightmare for one their most profitable brands.
Attacks on the ICs are probably possible if corners are being cut. Purely speculation but if this was the case I would be glad it was called out so fast.
How? I really doubt it as these ICs have no CPU, flash or anything else that could be programmed externally. Here's a random example of how one look like: http://www.ti.com/product/bq24232ha/datasheet
PSP was hacked through the battery. Basically Sony had a "magic" battery that would report certain ID, that they could use to flash any firmware they wanted. Well, someone figured this out, and as it turns out, you could use a hacked PSP(hacked through one of the existing exploits) to flash a different ID to the battery - from that point onward it would act as a "magic" battery and could be used to flash any PSP.
So yeah, the IC in the battery was nothing more than a tiny bit of memory containing the ID, but ultimately, that was the downfall of the whole PSP ecosystem, as you could use a hacked battery to reflash any PSP, regardless of its firmware, so sony had no way of patching this.
I' not aware of this particular case, but over-the-shelf charging IC is definitely not enough to accomplish that. That sounds like a wider, system problem. Anyway, that's an interesting story, I'll check it - thanks.
"This is a battery with its serial changed to 0xFFFFFFFF. When a PSP battery serial number is changed to 0xFFFFFFFF, or unreadable, the PSP boots the IPL from sector 16 on the physical drive (the Magic Memory Stick). This unlocks the service mode of the PSP and launches the IPL from the Memory Stick (instead of from flash0). A regular battery can be made into a JigKick via hardware or software methods."
>these ICs have no CPU, flash or anything else that could be programmed
All my laptop and phone batteries, if you view the battery info, have the name and date of manufacture, cycle count and approximate percentage remaining. This info is stored in the battery, not the device, so there must be memory and something like a processor.
>Macbook batteries ship with a default unseal password (0x36720414). This was found
by reverse engineering a Macbook battery update. On Macbook batteries, the full
access mode password is also hardcoded and default (0xffffffff).
>The actual firmware is machine code for a CoolRISC 816 8-bit Microprocessor. This
was found by Dion Blazakis by googling some opcodes from the end of the firmware.
Texas Instruments considers this information proprietary and would not reveal it.
>I this work, I provide API functions which can be
used to communicate with the battery. This allows the ability to make arbitrary
configuration changes as well as dumping of the data flash and instruction flash. I
provide IDA Pro scripts to disassemble the machine code from the firmware. We
provide a way to disable the firmware checksum as well as to make arbitrary changes to
the smart battery firmware. Due to the nature of the Smart Battery System, changes
made to the smart battery firmware may cause safety hazards such as overcharging,
overheating, or even fire.
Which sounds quite promising if you want to make Stuxnet like malware to toast peoples Macs.
These battery controllers can do stupid things if misconfigured and are at least partly controllable by the OS, maybe you even can flash firmware. If there is no additional "dumb" circuitry enforcing some limits to protect the battery there may be cases where it is possible to do real damage.
In theory the battery controllers are airgapped or at least communicating with the host via a very limited protocol such that they're unlikely to have "remotely" exploitable vulnerabilities.
In practice probably nothing. It probably will happen sooner or later, and then manufacturers will start taking security of this kind of thing seriously.
Given that I've seen "battery driver updates" in Windows Update for a few years now, I seriously doubt the security of that airgapping, at least in laptops.
> When several Samsung Galaxy Note 7 smartphones spontaneously exploded in August, the South Korean company went into overdrive. It urged hundreds of employees to quickly diagnose the problem.
> None were able to get a phone to explode.
What exactly does that process look like? I'm imagining a bunch of QA techs in a warehouse hoping for failure.
Maybe lots of extreme cases, like wrapping up the phone to try to overheat it, lots of testing of wireless charging (which seems to be a link between known cases), tests with various cables/chargers, maybe trying to get the phone wet...
This sounds like a good strategy for an unethical competitor. <100 devices and only a few uncorroborated reports but drastic images are enough to trigger this?
There are presumably plenty of ways to make a LiIon fail dramatically if you just know where to push or apply some directed heat. Ifixit has that covered I think.
Not even very expensive or difficult to do. A small group of people could coordinate such a thing pretty easily.
You can see a single opinion has taken over the internet where everyone is convinced any Note 7 in existence is a ticking time bomb that can kill you at any moment.
>You can see a single opinion has taken over the internet where everyone is convinced any Note 7 in existence is a ticking time bomb that can kill you at any moment.
Am I getting this right? We should be OK with 1 in 20,000 odds of a phone exploding in our hands or burning down the house?
Because that's what 97 in 2 million amounts to.
Besides, that's just in the first couple months of release. Can you guarantee that a device run with a known, faulty, battery issue, wont result in more explosion as it gets used for more time?
It's not like "if you switched it on for the first time and it didn't explode it never will".
If anything time (and wear on battery etc) could be a compounding factor...
There were less then that many shark attacks last year. People are still terrified of sharks.
Not a perfect analogy but people are terrified of things that are incredibly unlikely. It's legitimately hard for a phone manufacturer. 0.00001% is too many explosions if you sell a couple million phones.
There have been roughly a hundred incidents so far. The phone was out for about a month. Assuming first year sales would have been quadruple first month sales and the same rate of burnouts over the year, that's 100 * 4 * 12 = 4,800 incidents in a year.
One of these incidents happened in a car on the road and another happened on a plane. One happened while a child was holding it and another while the owner was asleep causing smoke inhalation injuries. Taking those as 'aggravated circumstances' we could expect to see 4 * 4 * 12 = roughly 200 cases in aggravated circumstances.
Looks like Samsung has to foot the 5+billion dollar bill to come up with a solution.
Are these known to just affect the snapdragon US version from whatever samples Samsung has been able to obtain? I'm reading that some cases are refusing to allow Samsung to investigate the phone.
My opinion is that putting the radio in the SoC is a big mistake. Qualcomm kraits are one of the main points Samsung has no control over, and it is not hard to imagine a scenario that could cause issues if they can't test it.
It's a conundrum: I would absolutely buy a Galaxy Note 7 when it's been out a while (given that Europe is considered a second class market, this happens automatically), but I would not want to buy it as an early adopter at the moment.
However, this mindset now also extends to a potential successor, so I don't think Samsung's strategy will play out quite as they think. The successor will have to be rushed even more.
I love the image that is conveyed of 'feature overload' being the cause of explosive devices. As though we may have reached some sort of technological wall against which all increases in device or feature richness lead to explosions
Well it's can be called an overload when you have components interacting with each other in a tiny space and add to that not giving people enough time to test the design. Obviously (imho) more time for testing and QA would have solved the issue.
The article was interesting and provided some information (on the battery suppliers, not being able to reproduce the problem) and perspective that I wasn't aware of before.
But the headline seemed inaccurate though. I read this article twice, and I still finished with the feeling that it didn't really answer the question in the headline. None of the interviews with identified or unidentified sources answered this particular question with any details. So it's a management decision because even the replacement phones had the same issue. Anybody following the news could've guessed that.
Though I despise some of Samsung's shameless tactics in the design copying department, I feel bad for the company and the employees who worked on this device. With the pressure of all the media coverage around, not knowing the reason for the failures would've been a terrible blow to morale and motivation. Hope the middle and lower level employees don't face the brunt for what could (still) be an impossible problem to figure out and solve.
I remember testing a wireless charging kit for my S4 while in China. The charger itself was rated 5V 500mA, but when I put my phone on it, the Ampere app reported 1900mA charging current and the screen started showing strong, wave-like distortions. I'm only glad the phone survived the experience.
They didn't explain why Samsung went to this crazy step.
First of all, it's a sign of poor Korean management leadership. Their testing and engineering departments still couldn't reproduce the errors, but pressure from upper management was so high that they went berserk, completely distrusting their engineers.
First of all, you cannot take such a dramatic step, losing face, when you have no idea what went wrong, and what caused the problems. Apparently it wasn't the battery supplier. So it's more likely the battery controller. Which would be trivial to fix once identified and analyzed. Call back, yes. But burn it, crazy. Typical Korean drama. Managers shouting and going crazy.
> First of all, you cannot take such a dramatic step, losing face, when you have no idea what went wrong, and what caused the problems.
They already lost face with a bit mismanaged and not so customer friendly replacement procedure and then again delivering dangerously faulty items. It's reasonable from a marketing perspective to end the disaster once and for all. "Oh you own a Note 7? Are you sure it's the really really really not exploding one?"
> So it's more likely the battery controller. Which would be trivial to fix once identified and analyzed.
And if it's not? Is e.g. the possibility of the too dense packaging damaging the battery completely ruled out?
Without knowing the problem, you cannot take such a step killing off your whole product.
How long does a normal non-korean company need, to take the next step? Longer.
How long did Toyota need to announce the next step for its recall crisis? Did Toyota kill their flagship products for the floor mat, accelerator pedal or hybrid anti-lock brake software problems, which killed people?
Dense packaging is also possible. CPU temp. or cooling also. Many possible reasons. That's why you analyze it first, and then announce a plan to fix it. Almost everybody had such a phone.
> Without knowing the problem, you cannot take such a step killing off your whole product.
Maybe they really only killed the name and plan reuse most components in a specification wise very similar soon to come successor model. If it's not a software problem but dense packaging or cooling they have to redesign a lot of things anyways.
> How long did Toyota need to announce the next step for its recall crisis? Did Toyota kill their flagship products for the floor mat, accelerator pedal or hybrid anti-lock brake software problems, which killed people?
A car is also completely different kind of product, you can change a lot of things without completely taking it into bits.
And how do you correct a fatal design flaw in a Phone without a recall? How do you do replacements without having a fixed design?
Samsung did the right thing here.. the fact that other companies don't isn't the problem. And, yeah, it probably was marketing driven as much as anything else, but I'd rather see this as a decision from a company than some of the deny, deny, deny tactics that other companies seem to take as a default strategy.
Have any of the phones actually exploded as in gone bang and ejected stuff outwards quickly. All the examples I've seen seem to have overheated and caught fire. I'm guessing exploded just sounds more dramatic in reporting?
70 comments
[ 5.5 ms ] story [ 294 ms ] threadhttp://articles.latimes.com/1985-11-14/local/me-2370_1_aids-...
https://en.wikipedia.org/wiki/Chicago_Tylenol_murders
Edit: I commented before reading the article and just noticed that this incident was mentioned in the article itself.
Even though the percentage is vanishingly small, Samsung cannot be associated with exploding phones.
Even airlines warn against Samsung Note 7s specifically during preflight briefings. They're amputating part of their business to stop contagion to the rest of their brand.
That's what I'd do, but then again I'm not a highly paid marketing executive.
I thought so too. I meant it's just been three explosions, or so I heard. However this article claims 97 incidents with breakdowns on overheating vs burning. It seems pretty significant. I mean, not statically against devices in the wild, but it takes a relatively few amount of safety incidents to question an entire product line.
Far more than 3. It's already more than 3 on the "replacement" "fixed" 7s -- it's dozens for the first version, including in planes, while driving, etc.
There was a rash of exploding battery cells in the 2006-2007 era where Apple even issued a recall. At the time Apple was still using off-the-shelf cells and hadn't started to make their own batteries. This affected many other vendors using the same suppliers.
Even Boeing's planes were not immune to battery problems, with the 787 famously suffering some serious issues, including fires.
Battery technology is something that's very hard to get absolutely right, to render something safe enough it won't explode. You want a lot of "sigmas" in your quality to avoid any serious defects.
I don't know what Apple does to their batteries, but the sorts of abuse they can apparently endure without catching fire is remarkable: https://www.youtube.com/watch?v=iy5kAY5SqEk
>> iPhone 6 (/Plus) has exploded quite a few times actually (not pasting links here, a simple search shows many).
And the iPhone 7 too : http://www.express.co.uk/life-style/science-technology/71958...
Most of the other phones are not exploding (other brands/Samsung's other phones) in significant numbers, so Note 7 is a one off case.
If it's a manufacturing problem, that's one thing, but I have a sneaking suspicion this could boil down to software. You can measure your manufacturing process very closely, but testing embedded software, such as in the battery itself, is a total nightmare by comparison.
Because engineers are OK if "only" 1 in 100,000 devices explodes and hurts someone?
Remeber the flying coffin? https://en.wikipedia.org/wiki/Lockheed_F-104_Starfighter#Saf...
It's still high enough to indicate a general manufacturing problem. To ditch it completely may mostly be a marketing decision but to put production on hold for some time to investigate and really fix the root cause seems to be reasonable. It may also help internal processes making sure that no additional item from a possible faulty batch ever gets sold accidentally. Maybe we will see a similar device in a few months, maybe branded differently, maybe with a redesigned casing, maybe some marketing campaign showing how well they've protected the battery this time.
One of the more comical attendees turned it into a joke, something like '... If you would like to stow your larger carry on, please come to the counter.... If you have a Samsung Galaxy 7, please place it in the garbage prior to entering the aircraft...'.
It was a crowded flight with a long line and already delayed. He was just trying to lighten up the atmosphere, and everyone did laugh, so it was clever.
For Samsung, this is a marketing nightmare for one their most profitable brands.
So yeah, the IC in the battery was nothing more than a tiny bit of memory containing the ID, but ultimately, that was the downfall of the whole PSP ecosystem, as you could use a hacked battery to reflash any PSP, regardless of its firmware, so sony had no way of patching this.
http://www.krizka.net/2008/02/10/what-is-pandoras-battery/
http://www.instructables.com/id/PSP-Hacking-Guide/step2/Pand...
https://geekindisguise.wordpress.com/tag/psp-pandora-battery...
"This is a battery with its serial changed to 0xFFFFFFFF. When a PSP battery serial number is changed to 0xFFFFFFFF, or unreadable, the PSP boots the IPL from sector 16 on the physical drive (the Magic Memory Stick). This unlocks the service mode of the PSP and launches the IPL from the Memory Stick (instead of from flash0). A regular battery can be made into a JigKick via hardware or software methods."
Cool dude.
All my laptop and phone batteries, if you view the battery info, have the name and date of manufacture, cycle count and approximate percentage remaining. This info is stored in the battery, not the device, so there must be memory and something like a processor.
Edit: There's some info here "Battery Firmware Hacking Inside the innards of a Smart Battery" https://media.blackhat.com/bh-us-11/Miller/BH_US_11_Miller_B...
With helpful advice like:
>Macbook batteries ship with a default unseal password (0x36720414). This was found by reverse engineering a Macbook battery update. On Macbook batteries, the full access mode password is also hardcoded and default (0xffffffff).
>The actual firmware is machine code for a CoolRISC 816 8-bit Microprocessor. This was found by Dion Blazakis by googling some opcodes from the end of the firmware. Texas Instruments considers this information proprietary and would not reveal it.
>I this work, I provide API functions which can be used to communicate with the battery. This allows the ability to make arbitrary configuration changes as well as dumping of the data flash and instruction flash. I provide IDA Pro scripts to disassemble the machine code from the firmware. We provide a way to disable the firmware checksum as well as to make arbitrary changes to the smart battery firmware. Due to the nature of the Smart Battery System, changes made to the smart battery firmware may cause safety hazards such as overcharging, overheating, or even fire.
Which sounds quite promising if you want to make Stuxnet like malware to toast peoples Macs.
In practice probably nothing. It probably will happen sooner or later, and then manufacturers will start taking security of this kind of thing seriously.
> None were able to get a phone to explode.
What exactly does that process look like? I'm imagining a bunch of QA techs in a warehouse hoping for failure.
There are presumably plenty of ways to make a LiIon fail dramatically if you just know where to push or apply some directed heat. Ifixit has that covered I think.
You can see a single opinion has taken over the internet where everyone is convinced any Note 7 in existence is a ticking time bomb that can kill you at any moment.
Whereas the truth is?
Because that's what 97 in 2 million amounts to.
Besides, that's just in the first couple months of release. Can you guarantee that a device run with a known, faulty, battery issue, wont result in more explosion as it gets used for more time?
It's not like "if you switched it on for the first time and it didn't explode it never will".
If anything time (and wear on battery etc) could be a compounding factor...
Not a perfect analogy but people are terrified of things that are incredibly unlikely. It's legitimately hard for a phone manufacturer. 0.00001% is too many explosions if you sell a couple million phones.
One of these incidents happened in a car on the road and another happened on a plane. One happened while a child was holding it and another while the owner was asleep causing smoke inhalation injuries. Taking those as 'aggravated circumstances' we could expect to see 4 * 4 * 12 = roughly 200 cases in aggravated circumstances.
Are these known to just affect the snapdragon US version from whatever samples Samsung has been able to obtain? I'm reading that some cases are refusing to allow Samsung to investigate the phone.
My opinion is that putting the radio in the SoC is a big mistake. Qualcomm kraits are one of the main points Samsung has no control over, and it is not hard to imagine a scenario that could cause issues if they can't test it.
However, this mindset now also extends to a potential successor, so I don't think Samsung's strategy will play out quite as they think. The successor will have to be rushed even more.
Think Chineseium, but a bit further south.
https://www.washingtonpost.com/news/the-switch/wp/2016/09/29...
But the headline seemed inaccurate though. I read this article twice, and I still finished with the feeling that it didn't really answer the question in the headline. None of the interviews with identified or unidentified sources answered this particular question with any details. So it's a management decision because even the replacement phones had the same issue. Anybody following the news could've guessed that.
I remember testing a wireless charging kit for my S4 while in China. The charger itself was rated 5V 500mA, but when I put my phone on it, the Ampere app reported 1900mA charging current and the screen started showing strong, wave-like distortions. I'm only glad the phone survived the experience.
First of all, it's a sign of poor Korean management leadership. Their testing and engineering departments still couldn't reproduce the errors, but pressure from upper management was so high that they went berserk, completely distrusting their engineers.
First of all, you cannot take such a dramatic step, losing face, when you have no idea what went wrong, and what caused the problems. Apparently it wasn't the battery supplier. So it's more likely the battery controller. Which would be trivial to fix once identified and analyzed. Call back, yes. But burn it, crazy. Typical Korean drama. Managers shouting and going crazy.
They already lost face with a bit mismanaged and not so customer friendly replacement procedure and then again delivering dangerously faulty items. It's reasonable from a marketing perspective to end the disaster once and for all. "Oh you own a Note 7? Are you sure it's the really really really not exploding one?"
> So it's more likely the battery controller. Which would be trivial to fix once identified and analyzed.
And if it's not? Is e.g. the possibility of the too dense packaging damaging the battery completely ruled out?
How long does a normal non-korean company need, to take the next step? Longer. How long did Toyota need to announce the next step for its recall crisis? Did Toyota kill their flagship products for the floor mat, accelerator pedal or hybrid anti-lock brake software problems, which killed people?
Dense packaging is also possible. CPU temp. or cooling also. Many possible reasons. That's why you analyze it first, and then announce a plan to fix it. Almost everybody had such a phone.
Sure you can. In fact, they just did.
Maybe they really only killed the name and plan reuse most components in a specification wise very similar soon to come successor model. If it's not a software problem but dense packaging or cooling they have to redesign a lot of things anyways.
> How long did Toyota need to announce the next step for its recall crisis? Did Toyota kill their flagship products for the floor mat, accelerator pedal or hybrid anti-lock brake software problems, which killed people?
A car is also completely different kind of product, you can change a lot of things without completely taking it into bits.
Samsung did the right thing here.. the fact that other companies don't isn't the problem. And, yeah, it probably was marketing driven as much as anything else, but I'd rather see this as a decision from a company than some of the deny, deny, deny tactics that other companies seem to take as a default strategy.