"We’re not trying to get into an argument with the security guys. They’re dealing with theoretical concerns." - Peter Hortensius
I'd say that someone having cracked out the password for the private key is a bit more than a 'theoretical' concern. This might be the most tone-deaf handling of a potential PR disaster so far this year.
This is cluebat level of ignorance. I want to apply the "don't ascribe to malice what can adequately be explained by ignorance" maxim, but I'm having trouble with the "adequately" here. Either they managed to live under a rock and completely ignore everything related to the Snowden revelations, or they're willfully dismissing it.
Such a pity, I was looking forward to getting an X1...
The price of "don't ascribe to malice" is that it's trivial to exploit. The most incompetent PR department in the world can have a strict adherent on a leash and barking to their tune inside of 5 minutes.
Courts of law have a good reason to hold high standards of evidence. For everyone else it's just an excuse for laziness. Not that I'm any better, I just don't insist on rationalizing it :-)
I'm now old and cynical enough to understand/believe that the world is run with laziness as the prime heuristic (with secondary heurisitc being "don't die/maintain current level of comfort").
I agree that it's trivial to exploit, but I choose to believe that, in the general case, people/entities are lazier than they are evil. That said, laziness can be in the form of "not taking into account the externalities", which can be indistinguishable from actual malice (which I define as knowingly and/or willfully causing harm).
I still think Lenovo's behaviour in this case is that form of laziness, though my comment above means I'm on the fence.
thoughtlessness and callousness are far more common sources of evil than either incompetence or malice strictly defined. "People who don't care if I live or I die".
I realize now I've been mixing up "incompetence" "ignorance" and "laziness". I used "ignorance" in the maxim in my first comment where it's actually "incompetence", and then discussed "laziness" in my followup comment. Sorry.
FWIW, I agree with you: I put thoughtlessness and callousness fall under the laziness umbrella.
I tend to suspect you're right about this being a case of malicious laziness. Once the engineers started protesting, middle management decided it was easier to feign ignorance of the problem and push on than to come up with a more legitimate value-add. Laziness-driven, but (I suspect) also entirely malicious in the sense that if you had given anyone in the room a choice between receiving a laptop with their crapware and a laptop without, they would have chosen the latter without the slightest bit of hesitation.
Then again, I may be underestimating management's ability to drink their own kool-aid.
My concern when making a purchase is whether the purchase will be good for me. I would definitely wipe whatever laptop I purchased, so this story is interesting and embarrassing for Lenovo but has no effect on my own purchases which are driven by what will function well for my preferences (like good Linux support and not requiring me to use a trackpad)
Given the general sliminess in computer, phone and software companies, there is no "pure" option except to buy some ancient computer and never use it on the internet, like RMS. This isn't acceptable to me. I will buy what serves my own needs, and you can do whatever you want.
I've been eyeing Macbooks forever, but as far as I know Linux support has never been great, despite the prevalence of their hardware. System76 sells some good Linux-oriented laptops. Some of Dell's laptops are Ubuntu-certified. I had a good HP Elitebook via work about 6 years ago, but everything I've tried of theirs over the past couple years has been throw-out-the-window bad.
If Lenovo are willing to compromise user software for some perceived corporate benefit, what's to say that they're not going to compromise hardware, firmware, bootloaders, recovery tools, etc.?
I ask that from a Lenovo Thinkpad T520i, one of a half-dozen or more I've owned or used over 15+ years, and absolutely my preferred mobile hardware over that period.
> This might be the most tone-deaf handling of a potential PR disaster so far this year.
It's appealing to a common and sucessful strategy of dismissing the concerns of experts as the irrelevant waffling of a bunch of eggheads disconnected from reality. Lenovo are hoping their user base will pop "security researchers" in the same bucket as beachfront property owning SUV drivers place "climate scientists".
> This might be the most tone-deaf handling of a potential PR disaster so far this year.
I think you're assuming that the broader public shares the indignation of HN about this. On mainstream news sites, it's down under the 'technology' heading. It sounds like Lenovo is scrambling a fix that will remove the certificate. If that's out in the next day or two and they have a way to get most affected users to apply it, probably hardly anyone will get clearly 'hacked'. They'll keep playing the 'honest mistake' card, and it will mostly blow over.
Less technically inclined people will still buy Lenovo's lower range products, and I'm sure businesses will do the enterprise thing and continue to buy Lenovo, since they tend to be the definition of ossified. But what about power users? I think Lenovo (and ThinkPads in general) have a reputation as the power user computer for Windows and Linux users. Not the biggest hit to be sure, and nothing to get the attention of CNN, but surely Lenovo will lose a certain segment of power users.
They're losing favour among a - probably fairly small - segment of users. But they're probably calculating that they'd lose much more customer confidence by admitting that it's a big problem, especially before they've got a fix out.
In a couple of days, they might be much more contrite about this. But today, the PR department's number one job is to keep it from becoming a big story in mainstream media.
This just occurred to me: Lenovo is a Chinese company and this broke right at the start of the Chinese Lunar New Year (a Christmas-sized or arguably bigger holiday in Chinese culture). Obviously I can't prove this is having an impact, but it wouldn't shock me to find out that this contributed to the amazing PR tone-deafness and incompetence of their response so far. Of course, that wouldn't excuse any of the terrible decisions made earlier, but it would be an interesting wrinkle from the perspective of assessing their crisis-management.
Nearly every laptop is preloaded with crapware. Lenovo might have (briefly even) picked one of worst examples of it but I'm sure there are (or will be) examples of this from other manufacturers that has yet to be discovered.
It might now be the case going forward that Lenovo will be a better choice. They've been burned.
There's crapware, and then there's spyware/adware.
The shovelware that most vendors ship on their boxes is offensive, yes. It's annoying. It steals a little of my life each time I buy a new machine, because I have to take time to re-image the system or clean off the crap (my current HP Envy was particularly egregious in this waste of my time, in that the restore image didn't work, so I had to wait ten days to get a restore DVD from them, and had to pay them $15 for the privilege of being able to restore my system). But, none of this is comparable to installing spyware on your customers systems.
They keep making claims that it isn't spyware, but in a previous HN thread, someone was trivially able to find the tracking and re-targeting codes in the injected code. It is the definition of spyware, and even worse, it is broken in such a way that it enabled MITM attacks.
"It might now be the case going forward that Lenovo will be a better choice. They've been burned."
Have you read their statements about it? Every single one of them denies any wrongdoing. They believe it's just a "customers don't like this software" issue. They don't believe it is a "We have likely committed crimes against our customers", which is what it actually is, at least in jurisdictions that take citizen privacy at all seriously. (In the US the TOS click through probably protects them, because the US doesn't give a shit about privacy, but in some other countries it probably wouldn't.)
Of course. You make out like that's a significant detail. "Large company denies liability" is not a headline. Do you honestly think Dell, HP, or even Apple would say anything differently in a similar situation?
Based on the quantity and quality of the software pre-installed on every laptop I've ever owned, I'm not quite as convinced as you are that this is exclusively an issue that could only ever happen to Lenovo customers.
Sony shipped spyware, too. As has Samsung on their TVs. But, that doesn't mean it's OK, or that anyone should trust Lenovo because "they've been burned". Lenovo isn't the wronged party, their customers are.
I'm saying I would need to see a "mea culpa" from Lenovo before I would even begin to think about trusting them.
I never intended to imply that anyone should implicitly trust Lenovo now because this happened; such a point is so ridiculous I'm surprised you believe I was making it. I also never made that point that Lenovo is somehow the wronged party -- which is again is ridiculous.
Instead, given the general attitude of most PC manufacturers with regard to what is pre-installed, I don't think boycotting Lenovo would necessarily save you from this sort of issue in the future. You're just as likely to be burned by almost any one of them. This product wasn't even made specifically for Lenovo.
Perhaps this attention will make Lenovo more careful in the future. It equally might not. It might make other manufactures more careful. Or it might make no difference at all.
The weird part is, I've bought three Lenovos of my own and never had a problem with their preinstalled software. Some of it, like the ThiknVantage suite, can even be useful. Maybe it's because I buy from the Thinkpad line? Do they install less crap on Thinkpads than on IdeaPads or other laptops?
Yeah. I have a ThinkPad T530 and a Yoga Pro 3 and while it's not a fair comparison (mobile workstation vs. strange laptop/hybrid...thing) the software on the ThinkPad feels more sparse and out of the way while the stuff on the Yoga is pretty terrible and invasive.
The last time I bought a Thinkpad (which was admittedly ten years ago) it didn't have anything I'd describe as bloatware. The quality of some of it was lacking, but it was all useful.
I attempted to start a conversation here about ways to detect whether visitors to your own site are infected with the Superfish malware, and I was downvoted to the very bottom of the page.
Another person had discovered a method to automatically disable Superfish by placing a special <meta> tag on your page; within two hours of posting his discovery, Lenovo silently removed the disable ability: https://news.ycombinator.com/item?id=9076788
Deliberately preventing people from disabling Superfish doesn't seem like something a company "working to wipe Superfish app off of PCs" would do.
I don't believe a single word uttered by these snakes.
Well, they have no clue how to handle this. I just upvoted your other comment, and I suggest that anyone reading this does the same for it, my comment and chinathrow comment above. All three seem to have been bashed, and even if this might be a legit downvoting, I tend to think this is a bit much of a coincidence - let's show them it won't work.
Raise hell. If you have one of these machines, file a complaint with the FBI. At the very least, fill out their form: https://complaint.ic3.gov/default.aspx
While I am all for giving Lenovo hell, I'd highly recommend that people avoid voluntary interaction with US "law enforcement". They aren't really on your side, and this has been demonstrated more than once.
I'm aware that Microsoft is legally limited from restricting what vendors can pre-load into Windows but they really should provide a technical solution. For example an operating system option to revert to absolute stock. This would necessarily be different from reinstalling from recovery (which also reinstalls the crapware).
Hortensius says Lenovo failed in due diligence. How, exactly? The two reasons people are upset about Superfish are that it breaks web security and injects ads into pages. Hortensius doesn't believe the former, and the latter is the entire purpose of the software. The only failure he could mean, taking him at his word, is in not anticipating the backlash, because as he describes it Superfish works as intended.
From a corporate perspective, Hortensius approved the bundling of adware (so he's ok with it) from what he thought was a reputable third-party software company. He was wrong!
Double standards are also seen in their "Removal Instructions" post on their forum. When uninstalling SuperFish, it seems suddenly important to remove the root certificate...
"It is very important to delete the certificate even though the application itself has been removed."
These are the URLs on the malware peddler's server I examined to get an idea for how to detect whether their malicious payload injection has taken place:
Lenovo should just offer the unbiased, truthful and transparent choice between a clean image (with device drivers) and their usual crapware image. They can then easily gather metrics on what image is selected when customers order products. Just give customers a choice!
Sadly there is no money in the clean image option for Lenovo. Margins on Non-Apple PC hardware are now so razor thin its only through the pay to play software "value-add" can companies turn enough money to justify the whole expense.
Some articles posted with a picture of thinkpad which is non-affected products for this issue.
>>Lenovo Y50, Z40, Z50, G50 and Yoga 2 Pro models.
Above is announced from some sources and
>>Lenovo-branded devices sold between September 2014 and January 2015 through consumer online and retail stores, like Best Buy and Amazon.com, are likely affected by the Superfish adware
As usual bundling stuff together is anti-consumer practice. If we have laws against bundling then it would be better world because a lot of useless at best/malicious at worst stuff wouldn't be able to get money (like crapware no-one sane would order separately).
Here is something to start with:
"if you offer a product A bundled with product B you have to offer A alone for the price not more than A bundled with B"
while weak and not really addressing many issues with bundling it at least gets rid of the most blatant problem of malicious add-ons.
Then something like A for not more than price(A) + 1/2price(B) and now we are getting rid of a lot more useless stuff.
I can understand the motive, however flawed it may seem in hindsight. They were seeing dollar signs, happy shareholders, job security - so they took a chance that massively backfired.
Am I the only one who thinks they deserve a little credit for getting their CTO to publicly deal with the issue on the same day all this came to a head, including saying that a guy was literally sat coding out a removal tool right now and due for release tonight? That's a damn respectable feedback loop for a megacorp if you ask me, and we'd have a lot less to whine about here on HN if all mistakes were rectified so expeditiously.
Lenovo, Peter Hortensius, Yang Yuanqing: I'm writing this on a T520i laptop. It's one of a half dozen or more IBM or Lenovo Thinkpad products I've owned personally or used through work over the past 15 years.
And yes, while I'm running my own installation of Debian GNU/Linux, I preserved the Windows installation for (very) occasional use. Only under a VM, and not in two or more years that I recall. With this news I'm strongly inclined to wipe it
But pulling crap like this is a tremendous erosion of trust. In your products. In any Microsoft Windows installation (not that I trust these in any event). It's a tremendous hit to your own brand equity, as well as Microsoft's.
The sad truth is that there are few alternatives out there, and that there are plenty of other security risks. But you can solidly bet that as a consumer and IT director I'll explore the hell out of other alternatives before making my own or corporate purchases and/or recommendations.
53 comments
[ 15.3 ms ] story [ 1804 ms ] threadThe only thing that will change going forward is they're going to do more due diligence on their AdWare suppliers before agreeing to the deal.
I'd say that someone having cracked out the password for the private key is a bit more than a 'theoretical' concern. This might be the most tone-deaf handling of a potential PR disaster so far this year.
Such a pity, I was looking forward to getting an X1...
It's like the opposite of damage control.
Courts of law have a good reason to hold high standards of evidence. For everyone else it's just an excuse for laziness. Not that I'm any better, I just don't insist on rationalizing it :-)
I agree that it's trivial to exploit, but I choose to believe that, in the general case, people/entities are lazier than they are evil. That said, laziness can be in the form of "not taking into account the externalities", which can be indistinguishable from actual malice (which I define as knowingly and/or willfully causing harm).
I still think Lenovo's behaviour in this case is that form of laziness, though my comment above means I'm on the fence.
FWIW, I agree with you: I put thoughtlessness and callousness fall under the laziness umbrella.
Then again, I may be underestimating management's ability to drink their own kool-aid.
Given the general sliminess in computer, phone and software companies, there is no "pure" option except to buy some ancient computer and never use it on the internet, like RMS. This isn't acceptable to me. I will buy what serves my own needs, and you can do whatever you want.
I've been eyeing Macbooks forever, but as far as I know Linux support has never been great, despite the prevalence of their hardware. System76 sells some good Linux-oriented laptops. Some of Dell's laptops are Ubuntu-certified. I had a good HP Elitebook via work about 6 years ago, but everything I've tried of theirs over the past couple years has been throw-out-the-window bad.
I ask that from a Lenovo Thinkpad T520i, one of a half-dozen or more I've owned or used over 15+ years, and absolutely my preferred mobile hardware over that period.
It's appealing to a common and sucessful strategy of dismissing the concerns of experts as the irrelevant waffling of a bunch of eggheads disconnected from reality. Lenovo are hoping their user base will pop "security researchers" in the same bucket as beachfront property owning SUV drivers place "climate scientists".
I think you're assuming that the broader public shares the indignation of HN about this. On mainstream news sites, it's down under the 'technology' heading. It sounds like Lenovo is scrambling a fix that will remove the certificate. If that's out in the next day or two and they have a way to get most affected users to apply it, probably hardly anyone will get clearly 'hacked'. They'll keep playing the 'honest mistake' card, and it will mostly blow over.
In a couple of days, they might be much more contrite about this. But today, the PR department's number one job is to keep it from becoming a big story in mainstream media.
You say you do due diligence to make sure the software you include is secure... yet you miss on such blatant vulnerabilities.
"Not doing enough" only scrapes the surface.
It might now be the case going forward that Lenovo will be a better choice. They've been burned.
The shovelware that most vendors ship on their boxes is offensive, yes. It's annoying. It steals a little of my life each time I buy a new machine, because I have to take time to re-image the system or clean off the crap (my current HP Envy was particularly egregious in this waste of my time, in that the restore image didn't work, so I had to wait ten days to get a restore DVD from them, and had to pay them $15 for the privilege of being able to restore my system). But, none of this is comparable to installing spyware on your customers systems.
They keep making claims that it isn't spyware, but in a previous HN thread, someone was trivially able to find the tracking and re-targeting codes in the injected code. It is the definition of spyware, and even worse, it is broken in such a way that it enabled MITM attacks.
"It might now be the case going forward that Lenovo will be a better choice. They've been burned."
Have you read their statements about it? Every single one of them denies any wrongdoing. They believe it's just a "customers don't like this software" issue. They don't believe it is a "We have likely committed crimes against our customers", which is what it actually is, at least in jurisdictions that take citizen privacy at all seriously. (In the US the TOS click through probably protects them, because the US doesn't give a shit about privacy, but in some other countries it probably wouldn't.)
Of course. You make out like that's a significant detail. "Large company denies liability" is not a headline. Do you honestly think Dell, HP, or even Apple would say anything differently in a similar situation?
Based on the quantity and quality of the software pre-installed on every laptop I've ever owned, I'm not quite as convinced as you are that this is exclusively an issue that could only ever happen to Lenovo customers.
I'm saying I would need to see a "mea culpa" from Lenovo before I would even begin to think about trusting them.
Instead, given the general attitude of most PC manufacturers with regard to what is pre-installed, I don't think boycotting Lenovo would necessarily save you from this sort of issue in the future. You're just as likely to be burned by almost any one of them. This product wasn't even made specifically for Lenovo.
Perhaps this attention will make Lenovo more careful in the future. It equally might not. It might make other manufactures more careful. Or it might make no difference at all.
LOL. Yes sir! The internet is filled with people happy about bloatware...
I seriously wonder how much money they make off these bloatware providers to risk pissing off customers and devalue their brand.
It can't be that much can it?
Another person had discovered a method to automatically disable Superfish by placing a special <meta> tag on your page; within two hours of posting his discovery, Lenovo silently removed the disable ability: https://news.ycombinator.com/item?id=9076788
Deliberately preventing people from disabling Superfish doesn't seem like something a company "working to wipe Superfish app off of PCs" would do.
I don't believe a single word uttered by these snakes.
Now waiting for the new Dell XPS13? Or anyone has another option for a powerful ultrabook that goes well with the pinguin?
But they are trying that through the signature store.
http://marcrogers.org/2015/02/19/will-the-madness-never-end-...
"It is very important to delete the certificate even though the application itself has been removed."
http://forums.lenovo.com/t5/Lenovo-P-Y-and-Z-series/Removal-...
Didn't seem that important earlier today: https://web.archive.org/web/20150219151726/http://forums.len...
"files in user directory will stay intact for the privacy reason. Registry entry and root certificate will remain as well. "
https://paste.ee/p/y1RvZ
These are the URLs on the malware peddler's server I examined to get an idea for how to detect whether their malicious payload injection has taken place:
https://www.best-deals-products.com/ws/sf_code.jsp
https://www.best-deals-products.com/ws/sf_preloader.jsp
https://www.best-deals-products.com/ws/sf_main.jsp?dlsource=...
>>Lenovo Y50, Z40, Z50, G50 and Yoga 2 Pro models.
Above is announced from some sources and
>>Lenovo-branded devices sold between September 2014 and January 2015 through consumer online and retail stores, like Best Buy and Amazon.com, are likely affected by the Superfish adware
Has anyone got any additional info?
Here is something to start with: "if you offer a product A bundled with product B you have to offer A alone for the price not more than A bundled with B"
while weak and not really addressing many issues with bundling it at least gets rid of the most blatant problem of malicious add-ons.
Then something like A for not more than price(A) + 1/2price(B) and now we are getting rid of a lot more useless stuff.
Am I the only one who thinks they deserve a little credit for getting their CTO to publicly deal with the issue on the same day all this came to a head, including saying that a guy was literally sat coding out a removal tool right now and due for release tonight? That's a damn respectable feedback loop for a megacorp if you ask me, and we'd have a lot less to whine about here on HN if all mistakes were rectified so expeditiously.
And yes, while I'm running my own installation of Debian GNU/Linux, I preserved the Windows installation for (very) occasional use. Only under a VM, and not in two or more years that I recall. With this news I'm strongly inclined to wipe it
But pulling crap like this is a tremendous erosion of trust. In your products. In any Microsoft Windows installation (not that I trust these in any event). It's a tremendous hit to your own brand equity, as well as Microsoft's.
The sad truth is that there are few alternatives out there, and that there are plenty of other security risks. But you can solidly bet that as a consumer and IT director I'll explore the hell out of other alternatives before making my own or corporate purchases and/or recommendations.