433 comments

[ 4.7 ms ] story [ 311 ms ] thread
According to various reports, this Superfish adware uses the same certificate across Lenovo computers. It should be easy to grab the private key out of the proxy binaries. And then... all these computers are vulnerable to arbitrary HTTPS man-in-the-middle attacks. Uh oh.
You're assuming that the proxy is on the laptops, no?
Well, the other possibility is that Superfish is routing and MITMing all traffic through its own servers, which is arguably worse.
arguably? That's orders of magnitude worse
Well, I dunno. In one case Superfish can see all your data and store it on their servers, in the other case _anyone on the internet_ can spoof any site (as soon as someone extracts the key). Either way is pretty bad.

But proxying all traffic from all Lenovo laptop owners through a third-party server without someone immediately noticing a problem is just not feasible, so I think we can assume that's not what they're doing.

Are you sure? Android Chrome proxies all non-HTTPS traffic through a third-party server, by default. So it isn't like the traffic volume is impossible.
Yes but that's Google. I'd be surprised if Superfish had resources like that, or could generate that much traffic from their servers and not be noticed (by, say, Google). I could be wrong.
Superfish might have "benefactors" with deep pockets who want a scapegoat who won't squeal on them.
(comment deleted)
This is much worse than just installing adware. They install a web proxy which MITMs all web connections, including HTTPS by means of a pre-installed trusted root certificate.

The root certificate is the same across all installs, and the private key is present on the machine (necessarily, to operate the proxy): https://twitter.com/fugueish/status/568258997578371072

Someone will extract the private key in the next few hours, and then HTTPS will be basically completely broken for all Lenovo users -- anyone will be able to spoof any site to them.

Uninstalling the app does NOT remove the certificate: https://twitter.com/metsfan/status/568265468173107200

On the bright side, Firefox does not use the system certificates (it has its own list) and Chrome will no doubt push an update to block the certificate promptly.

Anyway to see if that certificate is on a Lenovo computer? Anyway to remove it? I bought a Lenovo laptop recently, and I was appalled at the amount of crapware that was installed. It's a wonderful laptop at a great price, just too bad about the software.
Check Certificate Management in mmc.exe (Add Snap-In).
Or just run certmgr.msc.
It should show up in the system certificates list as "Superfish, Inc.". I haven't seen it myself but search for #superfish on Twitter to see a lot of screenshots and such.
> It's a wonderful laptop at a great price, just too bad about the software.

Lenovo's hardware support for Linux is great so unless there's something keeping you on Windows switching to a good Linux distro usually works fine on these laptops.

Do you trust a hardware vendor that installs MITM stuff on your machine per default to keep the firmware untampered?

There is almost no machine out there running openly auditable code on all components.

So what? At least with the software part you remove a large portion of the risks. It's better to go half way than doing nothing about it, and hardware tampering for a company could be more risky since they would have to do mass recall if discovered.
I'm talking Firmware-tampering, which is rather risk-free and firmware patches are not unusual.
> Do you trust a hardware vendor that installs MITM stuff on your machine per default to keep the firmware untampered?

Adding dodgy userspace software is easy and remunerative for Lenovo ( lots of $$$ from the software vendor for 'bundling' ).

Tampering with firmware is hard, expensive and doesn't seem to offer compelling return on investment. What's the business case?

Screen rotation is borked on the Yoga, apparently.
A cloudflare developer (I think) has put a test site up here:

https://filippo.io/Badfish/

The idea is something like

  <img src="haveproblem.gif"
       onError="this.src='noproblem.gif'/>
where haveproblem.gif is signed with the superfish cert (so you'll get an error if your machine does not have it, triggering the onError JS).
Here's Lenovo trying to justify the presence of this software, naturally oblivious to the security implications:

https://forums.lenovo.com/t5/Lenovo-P-Y-and-Z-series/Lenovo-...

naturally oblivious to the security implications

Rest assured Lenovo was perfectly aware of the security and privacy implications of this feature from the beginning.

They merely try to sound oblivious because their laywers hope that will soften the legal and media repercussions.

Honestly, I think that's unlikely. This is far too sloppy to have been intentional. There are much better ways to implement a backdoor when you control the OS image. This is just incompetence, plain and simple.

Superfish looks like the kind of crapware that pays OEMs to include it in their bundle. Lenovo took the cash and didn't bother to review the code. Superfish, for its part, probably doesn't have the best and brightest engineers working for them. They probably tasked a junior programmer with working around SSL, who then committed the first solution that worked without ever thinking about security implications, and they shipped it.

Cannot see how this could possibly be true. Having been privy to OS bundling for products, I can assure you there is lengthy contracts, and negotiations, about exactly what is happening. You do no simply walk up to Lenovo and have your "software" installed into the OS without a very detailed contract and pay structure. There also looks to be js injected into pages, which is serving up the ads, and a comment about Lenovo [1]. Think about what the means. There was a project at this company, where they had meetings, project plans, testing to make sure it worked, and a very detailed idea of what was going on. Never mind all the ramping up capacity due to new Lenovo's boxes coming on-line. There is zero chance this was some low level junior programmer fly by night operation.

[1] https://news.ycombinator.com/item?id=9072542

Oh I'm sure they had lots of meetings about the contracts and pay structure, and they may have done testing to make sure it didn't break things, but apparently no one did a security review. Sadly, this doesn't surprise me that much.

If they did know about the problem, they could have fixed it. If the app simply generated a new key as part of first-time use, then it would just be run-of-the-mill crapware rather than a gaping security hole. Even if Lenovo has malicious intent, it would still have been in their best interests to do at least that, yet they didn't. Hence I assume it was incompetence.

but apparently no one did a security review

It doesn't take a "security review" to spot a gaping security and privacy violation like this.

Any engineer with even the slightest clue of how a browser and "the internet" works would have called this out during the first "How does this product work?"-presentation.

Let's not pretend Lenovo is staffed with monkeys.

“Never ascribe to malice that which can adequately be explained by incompetence.”

Remember stuff like this:

http://www.cryptofails.com/post/70059600123/saltstack-rsa-e-...

(Which, possibly unfairly, is one reason I'm leaning more towards ansible than saltstack to this day -- I mean, if stuff like that got through... what else, in more complex areas of the system?)

The problem in Lenovo's situation is, calling it incompetence is the real stretch. You could call Charles Manson incompetent saying he just didn't know what he was doing was wrong, but everyone knows he was just evil.

Never falsely attribute to incompetence what is actually ascribable to malice. You can't come in here with a straight face and say that no one at Lenovo considered the security risk of including this software. If it was considered and they pushed ahead with it anyway, that's malice.

I don't think anyone there thought/realized that they were including a backdoor usable by any number of third parties (by virtue of installing a mitm-cert, and giving away the key). And this case is much worse than any other crapware-by-way-of-oem than I've heard of. But given the amount of nasty stuff most vendors seem to install on systems -- it appears to me that no one really looks at what is installed, or gives much thought to the consequences.

It's negligent, and in this case probably criminally so -- and that might constitute "an evil" -- but I don't think this is the result of someone's overt intentional evil act. I don't think anyone actually did consider the security risk of this particular piece of software. Maybe I'm naive, but if nothing else, the risk of lawsuits/backlash seems too great in this case.

I don't like ads and bloatware, but I think calling them "evil" is diluting what "evil" means.

I might be wrong, of course. But I don't think any of the big OEMs does any real review of the crap that is installed on computers -- and I think forgetting to generate an unique cert/key on post-install/first run is an error -- not intentional. Deciding to install this kind of crap strikes me as a very poor decision -- but I'm still not sure I'd consider it evil. Evil would be using the Intel management co-prosessor to do something similar -- presumably then a clean install wouldn't help.

But that argument means either that these companies do not have a security team (we know they do), that the security team signed off on this (we know they wouldn't), or the security team raised the risk and management chose to ignore it. There's absolutely no option that says "no one ever thought of this risk", at least not in the world we live in. I've worked in enterprise security and I still work in the security industry. There is just no way that this software got approved to be put in a default install and had no review from the security department.

That's what I meant by invoking the opposite of Hanlon's razor. Sure, never attribute to malice what can be explained by ignorance. But my point is, you can't explain this one with ignorance. There is just no way that Lenovo has hired a security team that would do a review of this and say it looks fine, and no way a company the size and stature of Lenovo would not have a competent security team. The only logical answer is that this was raised as a risk and management chose to accept the risk.

I'm not saying they're evil (I used that word to describe Charles Manson), nor that their end goal was for users to be compromised. Merely that they had to know this was a bad idea, and they chose to do it anyway.

You may be right. I'm inclined to believe the provisioning team in Lenovo is understaffed, and that they don't really do much security analysis at all. So I believe their negligent, and that their process is negligent. But I'm open to the idea that I might very well be wrong about that. Either way, it doesn't speak very highly of what kind of quality one can expect to get when shopping Lenovo products.
I generally agree, but this is a situation that can be explained by either an embarrassing level of incompetence or a pretty minor amount of malice (or even indifference). So I'll assume malice until I see them own up to that much incompetence.
You're so optimistic it hurts

"Any engineer" means something in HN, but we're not talking about "people who read HN" levels of engineer here, don't be mistaken.

Some people that have had no or limited experience with software are assigned to software projects, and that's the issue with companies like Lenovo.

Operations the size of Lenovo have a fairly intense vetting process before a product goes to market.

I find it very hard to believe that no red flags were raised by any of the engineers, managers and especially lawyers who must have screened this "feature" for problems.

It seems more plausible that the problem was known from the beginning (it is by design after all) and Lenovo decided to risk it.

I doubt the usual lawyer assigned to this understands SSL and certificates well enough to say anything about it. They worry mostly about contracts, and this is a technical thing.
(comment deleted)
> Operations the size of Lenovo have a fairly intense vetting process before a product goes to market.

How does that go along with a gigantic fuckup like this? Ipso facto there was no vetting, otherwise this wouldn't happen. What did they expect, that this wouldn't come out, that this wouldn't damage their brand even further? If it was done out of malice it is still poorly vetted and incompetent malice.

Just repeat, “Never ascribe to malice that which can adequately be explained by incompetence.”

They probably didn't figure out that anyone would have a problem with this. For them, it's just a cool gimmick to get some money. That it is a gaping security hole which makes about 0.42 % of user population mad, probably never occurred to them.

Unfortunately, for the 0.42 % (that is us, reading this site, and people of similar interests) it will be hard going to explain to the next 4.2 % why this is so bad. The remaining approximately 96 % of population will stay largely uninterested.

Hopefully we .42 will inform our fellow 4.2ers when they come to us for advice when buying a new laptop/anything Lenovo makes. I don't think it will be so hard to explain it to them. They already know what adware is. Just mention it comes installed ready to track you. Always listening while you're visiting bank.com.
> Just repeat

Yea, read again. I claim that even if there was malice there necessarily was an element of incompetence present in that case as well.

> it will be hard going to explain to the next 4.2 % why this is so bad

Why? People aren't interested in exact details, that's why they rely on 0.42%. You can illustrate the magnitudes of moronity required to design some of their products and lack of respect for security by explaining that they approach those that are needed to drive a car which has chainsaw strapped on its steering wheel. This isn't mere buffer-overflows due to bad coding, these are comatose levels of stupidity.

My own experience makes me suspect the same thing. I used to work for a company that was, at the time, trying to develop a privacy-enhancing product (ironically enough...) which did something somewhat similar (although not on the size of this fuckup -- they'd be intercepting, but not tampering with, encrypted traffic, and storing encrypted private data).

Virtually everyone in the engineering team raised a flag when the imbec...uhm, the Product Manager came up with the idea. We pointed out that a) this burdens us with the responsibility of storing sensitive data which can, at least, have significant legal implications and that b) even if it's encrypted data, it may be a little hard to market a privacy device that works by uploading user data to our server as a first step without being transparent about the whole process. Oh, and c) that the data recovery mechanism he proposed (which involved storing the users' private keys on our servers as well, just in case they lost their precious little gimmick) was, in this case, entirely retarded.

The whole thing didn't even make it to Legal, because everyone in the decision tree just thought that since there's no plaintext data being stored, there's no potential for a lawsuit (and when we told the PM about Lavabit, he came back two hours later saying he Googled it and that we're covered since we're not an e-mail provider). The bright heads in Marketing weren't exactly sure about the whole transparency thing. They thought we should keep it simple and just tell people that their data is safely encrypted and be done with it, because end-users don't need to know about tech mumbo-jumbo like encryption keys and all that.

I don't work there anymore (thank God) and they haven't launched in the meantime, but when I left, they were basically working on implementing this clusterfuck.

I'm sorry I can't be more specific than this (for obvious reasons, I hope). The point is, however, that decisions as complex as these (there's a stack of paperwork thicker than the Osbourne-1 involved in preloading anything on a laptop) are made through an elaborate process, not made "by mistake".

Someone knew there was a problem. The problem may have ended up misunderstood or washed out along the decision chain (although I find that fairly unlikely), but someone, at some point, decided this was ok.

Once one vendor in your space says "we filter HTTPS traffic for nasty viruses!", it becomes a marketing weapon, and lots of customers think "well, why should I go with A when B protects me better?"
How many engineers do you think were in the "how it works" meeting?
I don't know, I've worked on some large government projects where things like this could have possibly slipped through because an engineer or two thought it was a clever way to workaround the issue. Granted they should have known and may have known but I'm not convinced they had to have known.
They probably tasked a junior programmer with working around SSL

I don't think I've seen a junior anything who was informed and insightful enough to write a network proxy, including SSL support, and the necessary certificate work.

How could you add mitm functionality by mistake?
Because you call it "enhanced functionality featuring cloud services", not a "man in the middle attack".

And calling it enhanced is not always an unreasonable interpretation. For instance, take the case of a cheap mobile phone with a very limited bandwidth. You can increase the end user satisfaction considerably if you move some of the functionality to a server layer so that when you browse, the things actually happen somewhere in a cloud and your phone is just displaying the result, without being the actual browser as seen by the site you visit.

Nokia did this with some of the cheaper devices, and I think it was quite OK. It comes down to how much you trust that party, of course, and how critical your communication is.

I think you give them too much credit. This was probably a decision made by a non-technical group without input from a technical group (e.g. Marketing goes and does something without even thinking of contacting Engineering), and whoever slipstreamed it into the factory image just followed instructions unquestioningly. This will likely result in an eventual retraction and apology, and internal process improvements being made to prevent such things from happening again. Such things will eventually happen again because large orgs are inefficient and individual employees are frustrated by inefficiency, so they'll work around the protocols. Rinse & repeat.
(comment deleted)
Someone has posted the actual script elsewhere in this thread [1]. Of particular interest is line 194:

  if (location.protocol === 'https:' && queryString.search(/dlsource=hdrykzc/i) !== -1) // Patch for Lenovo - do not run on https sites
So yes, it seems someone at Lenovo was security-aware enough to demand an exception for HTTPS. Unfortunately the fine folks at Superfish either didn't understand or didn't care.

[1] https://news.ycombinator.com/item?id=9072542

No, this is an example of the Lenovo sales / marketing people making distribution deals with dodgy third-party companies. The people who design the machines don't make the decision to ship MITM proxies on them.

I honestly don't know why Lenovo (and others) still make these third party deals. Just ship the machine with a blank OS, or install a vetted selection of open-source software (7zip, VLC, LibreOffice if they want). Just don't install crapware for the mediocre kickback it generates!

For low-end machines these bundling deals likely form a sizeable chunk of the profit margin. (I've heard eyebrow-raising numbers for e.g. the default browser spot.)
Yep. The other chunk results from the OEM's refusal to stick to any long term consistency in the components they spec in consumer lines of devices. In business lines, you will likely get a 6-12 month guarantee with a 6-24mo forecast showing exactly what is shipping with what (CPUs, GPUs, screens, hard drives, etc). With consumer lines, they change components & suppliers any time, for any reason.
>With consumer lines, they change components & suppliers any time, for any reason.

I always love when the same model (down to the part number) comes with a different configuration and board inside the case.

Interesting this appears to only be on the consumer grade laptops. I know at first glance I saw nothing relating to it on my W540 that I bought in November.
notice how they focus 3/4 paragraphs on "the technology"
"When using Superfish for the first time, the user is presented the Terms of User and Privacy Policy, and has option not to accept these terms, i.e., Superfish is then disabled."

Brilliant! It is behind a "Terms of User and Privacy Policy" text.

And it's rather useless if the rogue CA is already in your trust store :(
It's awful even ignoring the security implications.

> To be clear, Superfish comes with Lenovo consumer products only and is a technology that helps users find and discover products visually. The technology instantly analyzes images on the web and presents identical and similar product offers that may have lower prices, helping users search for images without knowing exactly what an item is called or how to describe it in a typical text-based search engine.

Wow, there are tons of images on twitter about this [1]. There is one where they MITM https://www.bankofamerica.com/ too [2]. Why the hell would they do this. Brutal.

[1] https://twitter.com/search?q=%23superfish&src=typd

[2] https://twitter.com/kennwhite/status/568270748638318593/phot...

I assume it's easier to MITM everything.
Incompetance probably. They didn't realise that it would be that much of a bad thing.
Hanlon's razor: Never attribute to malice that which is adequately explained by stupidity.

... I wonder if there's an MBA / capitalism version of this, centering around short-term profit at the expense of everything else.

Jumping at short-term profit over the people who trusted you is malice, in my book. Profit-uber-alles is not some thing that appears out of the ether--somebody has to do it.
The certificate technique they use dates back to at least 2010 (possibly only in add-on form back then?) See https://groups.google.com/forum/m/#!topic/mozilla.support.fi... for example. This causes other problems too: http://www.id.ee/index.php?id=37046 It's not alone in this behavior: http://kb.mit.edu/confluence/display/istcontrib/Programs+tha...
What's funny is that they have three apps for photo-based matching of products...and pets. They really are a "visual search" company, a CA start-up of 80-200 people according to LinkedIn... They just seem to have forgotten the "don't be evil" parts of their business model...
> They just seem to have forgotten the "don't be evil" parts of their business model...

That or maybe they are completely clueless about the security implications.

Remaining questions: Does the superfish proxy itself check the certificate of the site it's connecting to? One would hope, but that's also a pretty easy thing to screw up.

If it does, does it trust its own cert? Probably (certainly?), but if not, that would leave one in the curious (perverse?) position of being safer by using the proxy. superfish can mitm your connection, but nobody else with the key could.

It's most likely not hard-failing on cert errors, otherwise any website with a self-signed or expired cert would be unaccessible. So that means you just lose warnings (and thus the ability to detect another MitM) in your browser.
I'm curious what legal stance Lenovo customers have here - their secure HTTPS connections are being MITMed intentionally - surely that's hacking, or some national security violation?
It certainly seems like unauthorised use of a computer system, on the face of it.
It's a big company doing, so it's gonna be fine.
It's not just because they are a big company though. The "community", the industry and the government all share blame for the lack of liability for software.

Edit: It's pretty bad form to downvote new accounts becuase you disagree. Imagine if I didn't know about hellbanning.

Ask yourself what open source licenses, corporate EULAs and the NSAs defense have in common. The best hope here is that Lenovo explicitly promised someone something they didn't keep.

(comment deleted)
It actually depends whether or not the practice is directly or indirectly agreed to by the user in the Terms of Use, Privacy Policy or similar document. Now, it's likely that users do agree to it, but if the language in their policies wasn't broad enough to cover action like this, theoretically it would be a violation of the Computer Fraud and Abuse Act, as exceeding authorized use.
This won't hold for Germany though. There is a concept of surprising clause (überraschende Klausel) as well as the concept of an unethical clause (sittenwidrige Klausel). In this case I would assume that both would hold even if there is some clause in the EULA. The BigCo argument holds in Germany unfortunately as well...
Some EULAs basically say "you give permission for us to access and modify any data in your system"... this is the first example that comes to mind:

http://en.wikipedia.org/wiki/PunkBuster

These agreements could be summed up in 3 words: "we own you".

At least PunkBuster is spying for a relatively noble purpose: preventing cheating in online games. Cheating absolutely destroys the experience in multiplayer games and has killed many games.

This is spying with the sole purpose of spreading ads and making money.

So because a few people decide to cheat at a game they paid for, everyone who paid full price for the game is forced to install spyware which can and does modify files on your pc, take screenshots as you play the game, monitor your mouse inputs, keyboard, etc...?
I think that is fine, personally. Obviously others might not. You have to specifically agree to install/allow PunkBuster, and you can choose to play on servers that don't use PunkBuster. With Lenovo not only is there no opt-out, but you're not even aware of the adware and root CA installation.

The "spyware" only spies on modifications to the game client in any way and tries to detect non-human involvement, which of course includes inspecting the file system and RAM. In theory it could harvest irrelevant information from your hard drive or memory, but no reverse engineer has ever made such a claim to my knowledge.

Valve Anti-Cheat does very similar things, but is run by what many consider to be a trustworthy company, so not that many people take issue with it. If one trusts the company that distributes the spyware, it's not really a problem, in my opinion. If Valve were to ever violate that trust, it would severely harm their business.

I also strongly disagree with DRM, because it only harms other players while providing no benefits. In contrast, online cheaters can completely ruin the playing experience for online games, and have heavily contributed to the death of some games.

I also have no issue if people decide to cheat when in single-player mode. If you pay for the game you should be able to do whatever you want if you're not affecting others. It's only a problem when they're playing with other people over the Internet. PunkBuster and VAC only run when you're playing in online mode.

It's not fine because, as is the case with Superfish, this type of software leaves gaping security holes that blackhats can exploit no matter how noble the vendor is.
What security holes does PunkBuster introduce? Adware like Superfish and game client modification detection like PunkBuster are very different kinds of software. I do not support anything like Superfish.
"Adware is malware with a legal team"
> their secure HTTPS connections are being MITMed intentionally

of course they are - Lenovo customers have signed the agreement that this is ok when they started the machine the first time </sarcasm>

"National security" is such a fickle concept.

You can bet that if the NSA manages to use this to hoover up some tasty HTTPS, this scandal will be lauded as a big boost to "national security" behind the scenes, and nobody will be punished. For all we know NSA had a hand in engineering this.

Of course, if some government data is stolen as a result, then the whole thing will be thrown under the bus and deemed a threat to "national security".

I hope anyone who uses terms like "national security" does it in full awareness of what Orwell meant by newspeak and doublethink.

Lenovo is a Chinese company, so it's possible, but you'd think they're more likely to be responsible.
Isn't superfish (or is it Phish?) a US/Israeli company?

Some of the code inserted is pretty strange, including functions to checks for lenevo, bestbuy.com and isPayingCountry() with a list of country identifiers:

http://pastebin.com/AQqWirba

So apparently they work with some big companies, and I can't work out what the country check is for, perhaps for subsidiaries of a large customer?

The code you linked is nothing out of the ordinary as far as adware in Chrome plug-ins etc. go. For an example have a look at the source code[1] of "Awesome Screenshot"[2] which is used by ~1,4M users and also calls home to 7 different hosts[3]. This is just one of many many Chrome plug-ins that is injecting ads and Google encourages this[4]. It makes sense to limit injections to markets they can serve / are affiliates in.

[1] https://github.com/heyalexej/pretty-fucked-up/blob/master/ba...

[2] https://chrome.google.com/webstore/detail/awesome-screenshot...

[3] https://gist.github.com/mvirkkunen/89f61a06819530e48b53

[4] https://developer.chrome.com/webstore/program_policies#ads

have a look at the source code[1] of "Awesome Screenshot"[2] which is used by ~1,4M users and also calls home to 7 different hosts

Insanity!

The NSA doesn't need this amateur-hour backdoor. They surely have control of one or more genuine certificate authorities already.
Impersonating a CA is not transparent and risks losing that CA if anyone finds out it's forging certs. They probably can do that, but it's a risky nuclear option.

This is a transparent dragnet that can easily be blamed away, which has been shown to be much more preferable in the NSA's M.O.

I really don't agree. Every government has an official CA, and last time one was caught (France with fake Google certs IIRC), nothing happened at all. Most CAs are too big to fall anyway.
The sad thing is we don't need to invoke the big bad NSA here. There is absolutely positively nothing about this that suggests it is anything other than bog-standard SSL incompetence.

And to be clear, I mean, absolutely nothing. This isn't a slightly unlikely thing that still leaves room to wonder about "plausible deniability"... this is a thing that happens all the damned time and the NSA need at most sit back and passively reap the benefits, along with hackers and criminals.

Somebody somewhere wanted to get in on the advertising gig because it looks like free money. Their first attempt didn't work on HTTPS sites. Some techie was ordered to fix it. Said techie read a few things on a few sites and typed in the magic commands to "make it work" and probably literally didn't even know that they'd just annihilated security for all their users... they literally just knew that this made their software "work", and for them, pretty much the first time they clicked on to an HTTPS page and saw their own ads, the story ended. Ship it.

To a first approximation, nobody using SSL in some manner understands SSL.

It does seem like this is more of an amateur hour screw-up. It isn't beyond the NSA to plant developers that can insert backdoors on their behalf or set up front companies to sell vulnerable libraries but one would hope that they have enough sense not to leave cleartext passwords in a binary. Of course that could be an intentional misdirection so one never really knows.
The employers that I know of who do government work require that all computers/phones work is performed on be of certain manufacturers which are US companies, an issue like this is the exact thing they cite as the reason for not using foreign companies as providers of such hardware. So the chance of government data being stolen is minimal, so the chance of the US government caring much is unlikely. So I doubt this will wind up under that bus.
It should absolutely be illegal to do something like this.
I think what you meant to say is that the existing laws that make something like this illegal should be enforceable in a meaningful way against large manufacturers and retailers.
Mozilla discussion about what to do with the Superfish cert:

Bug 1134506 - Mark "Superfish, Inc." root certificate as untrusted in NSS

https://bugzilla.mozilla.org/show_bug.cgi?id=1134506

While it is akin to playing whack-a-mole, it's nice to see them seriously considering blocking this cert so users who get a theoretical update in Firefox would have it simply be removed. Granted Superfish could update and get around it but that would require effort and considering the PR nightmare Lenovo is going to be fielding I doubt they would do so.
Yea, this should get into the news which will hopefully help a lot.
Is there reason to believe that the same key is used on all machines?
>They install a web proxy which MITMs all web connections, including HTTPS by means of a pre-installed trusted root certificate.

That's the odd part of this. Browser plugins can modify the DOM (insert ads, change search results, etc) without proxying anything. So why do it? I wonder if they were fishing for an NSA contract to further monetize the installs.

The proxy works for all browsers with a single codebase.
Browser plugins are easy to wipe out. When dealing with a rather persistent malware a few months ago, it had inserted a legacy policy for a proxy in the Windows registry in a place not commonly checked by malware scanners. You turn off the proxy settings, but at every reboot it would come back and nothing seemed to catch it at the time. Malware can inject things in to the local group policy and other places that are not commonly checked, such as the root cert store, making them very likely to be missed by tech support.
I'm confused; if Firefox doesn't use the system certificates, shouldn't Firefox users have been seeing visibly broken HTTPS from day one?
It's not broken, because the Firefox certificate storage isn't empty when you install it. It includes the ones recognized by Mozilla.

https://www.mozilla.org/en-US/about/governance/policies/secu...

Sure, but I assume Mozilla doesn't recognize the Lenovo adware, so if all the web traffic is being routed through this proxy, shouldn't firefox have squawked?
Except if the adware just modify OS proxy settings, like madeofpalk mentioned. Firefox does not take those into account.
When taking Firefox into use, it imports the OS proxy settings, though. You get a warning but I guess about 99 % of people don't care about what that means.
Mozilla has its own proxy settings as well, independent of Windows Control Panel configuration, so a Firefox user appears not to be impacted by the whole thing at all.
It's not clear to me. Just a few minutes ago (and after your post) this appeared on mozilla discussion forum given by [1] above (will come back to credit this- didn't copy and don't remember (and can't see!)).

https://bugzilla.mozilla.org/show_bug.cgi?id=1134506

Down around 0200 PST 2015-02-19

EDIT: credit

[1] cpeterso https://news.ycombinator.com/item?id=9072642

OK, so they might have added also a Firefox plugin that infects the Mozilla trusted CA list as well.

I guess Firefox should block that plugin as malicious.

(comment deleted)
I thought that Chrome checks and reports that google.com certificate is a google issued certificate. How did this mitm attack not pop up massive warnings in chrome?
Chrome ignores Trusted Root Certificates when checking certificate pinning.
But doesn't that defeat the purpose? If a trusted Chinese certificate authority issues some certificate on google.com for China to perform MITM attack, and Chrome ignores anything signed by a valid root certificate, it will never report this attack. I thought the point of certificate pinning is precisely that only a single authority can sign a certificate for a website.
This situation is quite common in enterprise deployments [1], where HTTPS traffic is MITM-proxied through a central server to e.g. check for malicious content or other filtering.

If Chrome were to block unknown roots for pinned sites, these sites would become inaccessible because the MITM proxy is still active. That's certainly not desirable in a controlled enterprise environment, but the same would occur when blocking this 'Lenovo root'.

[1] http://it.slashdot.org/story/14/03/05/1724237/ask-slashdot-d...

No, the purpose of pinning is to stop a compromised CA from issuing their own www.google.com cert.

If someone installs a CA, Chrome will trust it. There's not much way around this: if someone has the capability to install a CA on your computer, they'd have the capability to modify chrome.exe to force acceptance of it.

Also, sometimes MITM'ing is desired. I'm doing it right now with Firefox and BurpSuite.

I think the problem is rather giving a false sentiment of security to the unsuspecting user.
Chrome could display a notice reminding users that it's an executable that can be compromised by other programs. But those other programs could also delete that notice.
I was thinking more something like an amber icon instead of green, which shows this connection is somewhat secure but there are problems detected.
"Someone will extract the private key in the next few hours, and then HTTPS will be basically completely broken for all Lenovo users -- anyone will be able to spoof any site to them."

Do you mean the proxy is remote? That is not the impression I have (otherwise having the private key locally makes no sense).

If it's local, then even with the private key extracted, and considering a lot of website force https nowadays, we should still have standard crypto between the lenovo computer and the website. EDIT: As long as the adware checks the website certificate AND doesn't trust it's own self-signed certificate in the store... yeah... a lot of ifs...

Anyway, thanks for the additional details, more helpful than "[...] the certificate allows the software to decrypt secure requests[...]", found in the article...

Having the private key means you can sign your own certificates to serve HTTPS with, so no MITM required.
> we should still have standard crypto between the lenovo computer and the website

Standard crypto using that website's certificate. Which could be legit. Or could be an attacker's certificate, signed with this Lenovo root certificate.

Some criminals are about to make a lot of money.

Not if the proxy checks the certificate of the site it's connecting to and doesn't trust it's own self-signed cert (there is no point in doing so if it's pure adware). But yeah... I have no idea what it does...
I honestly doubt that someone who was clueless and lazy enough to use the same self-signed certificate on all machines would put in the extra effort not to trust that certificate. Besides, the certificate is left behind after the software's uninstalled and no longer proxying connections.
Komodia, the company behind the tech contracted by the maker of SuperFish, actually (tries) to makes sure invalid and self-signed certificate do generates a warning in the browser. And then they password protect the private key with... the name of their company?!?

http://www.komodia.com/wiki/index.php?title=SSL_Digestor#Cer...

"Also the module tries to verify that the certificate is indeed signed by an approved signer, it will use the CA store of the browser used to verify that (for Internet Explorer the Windows store will be used, and for Firefox the NSS store will be used), if the certificate isn't legit, the created certificate will be created in a way it would raise an alert to protect the user."

A huge ugly hack...

(comment deleted)
Ironically, I've been MITM'ing my HTTP and HTTPS for over a decade with Proxomitron, and it's been quite useful:

http://en.wikipedia.org/wiki/Proxomitron

Interesting question to consider: what if the MITM was benevolent to the user? I.e. Lenovo included a similar ad-blocking proxy in their default installation? Would the public response have been as negative, or would it be considered to be a helpful addition akin to how most browsers now include popup-blockers?

In other words, are people more repulsed by the purpose (advertising)? Because I certainly think MITM'ing connections locally to remove ads a good thing... and with some devices like "smart" TVs apparently now phoning home and showing ads, I have no qualms about putting their traffic through a proxy to strip that crap out.

The issue here isn't so much the ads as it is being able to authenticate that the remote party is who you think it is – if your browser trusts the MITMed certificate, you no longer have the guarantee that your banking website is actually your banking website and nothing nefarious, as the page has been intercepted (maliciously or not) in-flight.

avast! was actually guilty of this a while ago (see https://lelutin.ca/posts/avast_conducts_MitM_attack_on_users...), and the article gives some good rationale why MITMing SSL at all without the user's explicit knowledge is bad.

if your browser trusts the MITMed certificate, you no longer have the guarantee that your banking website is actually your banking website and nothing nefarious, as the page has been intercepted (maliciously or not) in-flight.

The trust essentially moves from the browser to the proxy - while I don't know what Superfish does, Proxomitron definitely checks the certificate and pops up a warning dialog if there's something wrong.

why MITMing SSL at all without the user's explicit knowledge is bad

I think "without the user's explicit knowledge" is the key point here; if you install a security product then you somehow expect that it be able to inspect all your traffic for any maliciousness... as otherwise the "bad guys" will just make use of SSL to defeat that.

Presumably (hopefully!) when you installed Proxomitron, it generated a new unique private key for your own personal MITM.

Apparently Superfish ships from Lenovo with the same private key on every machine. So all a bad guy needs to do is extract that private key from one machine, and now they can MITM all the Superfish Lenovo machines from basically anywhere on the Internet.

It does come with its own certificate by default, with instructions for generating your own, but it doesn't trust that certificate for external connections; it uses a separate database of trusted roots which doesn't include the MITM certificate.

Has anyone confirmed the certificate validation behaviour in Superfish? I have a feeling it will be "none at all", which would be really bad...

It's all fine when it's you who is controlling the MITMing. In this case, Lenovo's malware does this without knowledge of the user and uses the same certificate on each machine, private key for which is embedded in said malware. That private key has probably already been extracted (or it will be very soon) - and at this point anyone can MITM your Lenovo machine by using that certificate.
Why did he expect to find the password in the clear in the memory dump? He indeed found it there, but why would one expect to?
the nature of writing a blog post ex post facto?
So, just a hunch that it would be a company name or something else that might be in the dump? There's no technical reason for the actual password itself to somehow end up there? A serious security flaw or something?
What I mean is, we are reading about it because it worked.

It's the lowest hanging fruit. I doubt he expected to find the password just sitting there, but since he did, here we are :)

But yes, keeping sensitive information hidden in plain text considered a security flaw.

Because it is needed to decrypt the key and as the program uses it, it must be in memory (at least at some time).
Ahh. That makes sense. So the malware itself is decrypting the certificate using the password.
He didn't find the password in the clear, he found the private key in the clear. He brute-forced the password.

I assume his reasoning for looking for the private key was similar to: this program creates a new certificate authority and installs it on this computer. In order to do this, it must have all necessary tools for doing so, including the private key it uses to create those certificates, in memory somewhere. Even if that private key is stored encrypted somewhere, it has to exist unencrypted in memory at some point to be used.

Read it again, he found the password in cleartext in the memory dump. From the blogpost:

> I tried the small dictionary john.dict that comes with John-the-Ripper, and it didn't find anything. But of course, I don't need a real dictionary. The password is probably also in the clear in the memory dump. I could just use the file super.txt as my dictionary! I tried this, but it was taking a long time, with 150k unique lines of text. It'd take many hours to complete. To speed things up, I filtered the list for just lower-case words

The password to the key must be in the binary - either in clear or encoded form and at some point it needs to be in memory in decoded form. Otherwise the binary could not decode the key itself. You could drop the passphrase immediately after decoding the key to make it harder for the attacker, but fundamentally all info to decode the key must be somewhere on the machine itself.

Trying all strings from the binary if any of them matches is a cheap and easy operation, so try it first, if it doesn't work use a more elaborate approach.

Imagine if the person you bought your house from told you "I've disabled all the locks on your doors and windows so that I can pop in from time to time and leave a fruit basket on your dining room table."
Wow. I just bought my first Lenovo product recently, a Q190. I will not be purchasing anything from them again.
Better yet, return it as defective.
Yeah this is really disappointing. Lenovo had become my 'goto' recommendation for people looking for a laptop.

Sure as hell not going to be doing that any more.

This reinforces my policy of buying laptops with the cheapest drive offered and replacing the drive with an SSD before the first boot. I run Linux anyway, so booting Windows has no value for me.
Can't you just write all 0's to the drive or just reformat it? Genuine question here, why would you need to physically replace the drive to ensure security when you can write to the whole thing?
Would have to re-write/re-flash the firmware as well.
What is it that the firmware can achieve? Is the firmware capable hijacking data, communicating with the NIC and transmitting data? Or is it somehow injecting harmful code? I feel like I'm missing something here.
The drive firmware can change the bits going to/from the drive, no?

For example, it could binary-patch (either at write time or read time) your kernel image on disk to communicate with the NIC, etc...

Ripped from yesterday's headlines ...

  ... rewrote the hard-drive firmware of infected computers—a
  never-before-seen engineering marvel that worked on 12 drive
  categories from manufacturers including Western Digital, Maxtor,
  Samsung, IBM, Micron, Toshiba, and Seagate.

  The malicious firmware created a secret storage vault that survived
  military-grade disk wiping and reformatting, making sensitive
  data stolen from victims available even after reformatting the
  drive and reinstalling the operating system. The firmware also
  provided programming interfaces that other code in Equation
  Group's sprawling malware library could access. Once a hard drive
  was compromised, the infection was impossible to detect or remove.
http://arstechnica.com/security/2015/02/how-omnipotent-hacke...
That appears to be the act of a nation-state though. I don't really sweat those, because I'm pretty sure if the NSA really wants in to my machine, I can't stop them.
They don't want in to just your machine though, they want a backdoor in to everyones machine, by default, without cause.
I'm not saying it is acceptable or that it doesn't matter. Just that, when it comes to my own personal computer, it isn't worth worrying about.

I have a lot of friends who haven't figured out the whole security-as-a-spectrum thing, and they spend a lot of time giving themselves grey hairs over adversaries that 1) they can't beat, 2) aren't worth beating, and 3) don't care about them anyway.

> with the cheapest drive offered and replacing the drive with an SSD

I expect the "cheapest drive" is not an SSD.

hence "replacing the drive with an SSD"
I don't understand the downvotes. The gp probably asked why not just zero-out the bytes. Sure, there's the firmware modification issue. But what I was responding to is why replace. This is the easiest option.
Post says "buy the laptop with the cheapest drive; then replace that cheapest drive with an SSD".

You said "the cheapest drive is not an SSD".

The point is to minimise the money spent on the drive supplied with the machine because you're not going to use that drive, you're going to throw it away.

You also have to reflash all firmware with known-trusted versions using a known-trusted reflasher to be safe.

... and replace the CPU with one that is known not to have backdoors. You'll have to craft it from Silicon yourself, though, because there aren't any available for sale anymore.

This is what it looks like when people don't recognize that security is a spectrum.
I recently did this exact thing -- bought a Lenovo laptop with a 5400 RPM disk drive, and immediately popped it out and replaced it with a Crucial MX100 SSD. Installed Linux, it works great :)
or a mac. Apple will do something like this when hell freezes over.
Was just about to purchase a lenovo... although I would have wiped it and installed linux immediately this has caused me to look elsewhere. when will companies learn this kind of behavior is toxic to their business?
Unfortunately a very small proportion of potential customers are going to hear or care about this... it's about as toxic to their business as stepping in some stinging nettles is toxic to me.
unfortunately you're completely correct.
Unfortunately they have no competition in terms of a quality laptop to run Linux on. None of the competition offers similar features as my current x230 or the x250 I'm probably going to pick up later this year. If you could recommend a replacement that has a good keyboard, trackpoint, 12+ hours of real battery life, i7, etc. I'd be happy to hear about it.
I don't know; if this gets into the news cycle (which it should), I think it will be a huge problem for lenovo. The people buying one of these to run Linux likely already understand the implications and are reading about it now. The rest of the consumer base need only hear "someone can intercept your banking password" and they will take notice.

There has been an uptick in computer security related news stories lately. I think the tide may be changing, albeit slowly.

First thing I also do on a new PC is reinstall the OS from scratch and get rid of all the preinstalled shit.
Your strategy works only if you have a clean copy of the OS or you buy one (since the thread is about Lenovo I assume you are talking about Windows). Typically, a new PC doesn't come anymore with a copy of the OS, but with a hidden recovery partition that will basically let you do a factory reset (meaning all the crap will show up again).
Microsoft itself has provided Windows installation media for download since Windows 8, including Windows 7 media. All you have to do is read your key off BIOS or the sticker.

And of course Windows 10 will be a free download.

Unless things have changed, usually the sticker key is only valid for a certain kind of media. E.g. VLK's only work with VLK images, retail keys only work with retail images...
Just recently installed Windows 7 Pro on a HP ProBook thing:

- looked up the Windows and Office license keys of the existing installation, using an utility

- download Windows 7 disk image from Microsoft and burn on a DVD

- take out the old disk with recovery partitions and installation with crappy bloatware

- put in a new SSD disk, boot DVD to install OS and install Office

- download and install HP specific drivers for peripherals (display adapter, fingerprint reader, wlan/3g, whatever)

- enjoy a relatively bloat-free Windows experience with improved battery life

I usually use a KMS key to install and then use the Windows+Pause dialog to change my product key to the key that is stored in my BIOS.
That's true, but in the past I've found that if I call Microsoft support and explain that I'm re-installing, they'll give me a new key over the phone.
Free? I thought that was only if you already had 7 or 8 installed.
Yes, Windows 10 requires having 7 or 8. Still free.
I think you can just extract the OEM Windows key from Windows, download a clean retail .iso of the same version and activate it with the OEM key. I'm pretty sure that worked with Windows 7, no idea about 8+.
Can someone with one of these laptops connect to https://www.howsmyssl.com/ and post what it says? I'm curious what cipher suites are used from the proxy to the real site.
It says it's "Probably Okay", even when I have Superfish's certificate enabled. (I have the program installed, but the cert sticks around.)
The site cannot detect that you have an extra root certificate lying around on your computer. If you visit the website without the Superfish program installed, you just evaluate the SSL settings of your browser.
First thing I did when I saw that URL was to run it through Qualys' SSL Labs test. Multiple issues, no forward secrecy, weak ciphers, grade set to 'C'. Oh, the irony.

https://www.ssllabs.com/ssltest/analyze.html?d=howsmyssl.com

I am guessing the Lenovo machines that are bought from the Microsoft Store are free of this, because of the Signature PC program, might be worth the extra cost if any and the trip there to get a crapware free machine.
(comment deleted)
I wonder what are the legal repercussions of this, can't someone sue them?
I think worse than that, I see criminal charges being brought up for this including fraud, theft, etc.
Theft? Seriously?
They were making money (tens of millions) from software illegally installed, so definitely.
Do you know what the word "theft" means?
Just found this: Spy agencies ban Lenovo PCs on security concerns (27th July 2013) - http://www.afr.com/p/technology/spy_agencies_ban_lenovo_pcs_...

"Multiple intelligence and defence sources in Britain and Australia confirmed there is a written ban on computers made by the Chinese company [Lenovo] being used in “classified” networks."

Another black eye if they knew about Superfish and didn't alert their citizens.
It is from 2013 though.
IIRC, that article is a fine combination of bullshit and technically correct. There is a "written ban" purchasing equipment from anyone not on the approved vendor list. Lenovo didn't ask to be on the list, they're not on the list, therefore they're banned. As am I. As are you.
I thought that had to do with the fact that they're a chinese owned company and if say the CIA makes a large order (or any order really) the chinese government might step in and force malware to be installed.
I can't see why it would matter, since literally every laptop is made in China already. Plus the vast majority of computer components.
Maybe it's too much of a risk if exposed for the manufacturing industry had they added a backdoor to a foreign customer's component without their knowledge?

As opposed to Lenovo agreeing to implement a backdoor? I'm not sure either.

Right, I remember this and when the Huwai stuff came out. The typical anti-western loudmouths said it was protectionism. Now the very same loudmouths are back-peddling and assuring us that this was a simple oversight and there's no way any of this could ever be tied to the CCP. Its incredible how anything that happens in the US is a NSA plot but a fucking MITM shipped on millions of chinese laptops? Oh just a mistake from a junior dev, nothing to see here guys.

I sometimes wonder if autocratic regimes are so image focused that they've seeded popular forums with stooges.

I'm surprised that this is just now news. I received complaints from people participating in our beta trial (http://sketchtogether.com) from as early as October 22nd, 2014 that our website was broken, and it was because of Superfish being installed on their lenovo laptops. When they uninstalled Superfish, our webpage started working again.

Superfish injected a line of code that referenced "sf_main.jsp" from a remote site into all webpages (including ours) that interfered with our code. Here's a pastebin of the sf_main.jsp javascript file it linked to: http://pastebin.com/bZFkfRd5 (I assume the linked code is not copyrighted, if it is, please let me know and I can take it down).

An all-new reason to use Content-Security-Policy.

How much you want to bet that thing is XSSable?

>An all-new reason to use Content-Security-Policy

Correct me if I'm wrong, but I don't think any amount of CSP will help you in this situation. They're MITMing traffic and thus can modify the CSP headers.

Fair enough, though I'd bet they aren't smart enough to have actually blocked the header. They apparently don't even support WebSocket.
Line 194 -- They customized their ad script for Lenovo. Making them entirely aware of what's going on...
Googling "hdrykzc" returns some interesting results...
For reference, it's safe to assume that code is under copyright, but don't take it down: this is almost classic fair use.
`https://www.best-deals-products.com/' sounds like the classic online store that will steal your CC :-)
I wonder how many people would find the domain name suspicious - I instinctively felt "this sounds scammy to me" when I saw that name, but can't quite explain exactly to someone else how I got that feeling. Perhaps the keywords "best", "deal" and "product" raised the red flags for me, and it's an instinct acquired by many years of being online.
If the company/website name consists entirely of SEO keywords, run?
Interestingly it is disabled for Google services (making the article's image irrelevant :). If this regex matches, `nofish` is set true, which disables superfish:

/^https?:\/\/(www|play)\.google\.(?!com\/analytics\/)/i

Also, if you add a <meta name="superfish" content="nofish"> tag, it gets disabled as well.

Possibly some agreement with Google, like the ones they tend to make with ad-blockers? (http://www.theverge.com/2015/2/2/7963577/google-ads-get-thro...)

That doesn't disable the part of Superfish that MITMs SSL connnections to sites - in fact, it obviously can't because that check can't even run until they've MITMed the connection and injected the code that includes those checks.
> (I assume the linked code is not copyrighted, if it is, please let me know and I can take it down).

it probably is, but by the look of things one can safely assume that they can fuck off

So, what you're saying is that people still use the hard drives that come with their laptops.

Interesting....

The article says that Superfish "injects third-party ads on Google searches." Does that include https://encrypted.google.com/ in Chrome and Firefox, or do key pinning and HSTS preloading successfully prevent that?

EDIT: According to another comment here, HTTPS connections in Firefox aren't affected because they don't use the system certificate store. But what about Chrome - do users see an error on pages with pinned keys, or is the proxy smart enough not to attack those connections? Or does it also disable Chrome security features like HSTS and key pinning?

Locally added CAs override pinning, so no it wont help.
I presume the next step is Adware installation on in the flash of the system's boot drive.
Interesting that the Superfish job page is looking for an iOS kernel hacker. And by "interesting" I mean "horrifying".
What other purpose beyond the development of drive-by installs of iOS rootkits can such a job position have in a company like that?! :(
As a Lenovo owner, I'm really pissed off, and offended. I feel violated. I just can't comprehend how they could think they wouldn't get caught at something like this. Especially with the current climate of the privacy movement in the US. This is bad, very bad for Lenovo.
I just wanted to echo your sentiments. I bought my T440p last year and have otherwise been reasonably happy with it (though not entirely, due to the iffy trackpad). Fortunately the first thing I did was replace the hard drive. Despite that, I'll never buy another Lenovo product. I have completely lost confidence in the company.
How to kill a brand in 1 easy step: do this.
Well, it seems as though this [superfish] is categorized as a virus on most websites. From their own description:

"Superfish Window Shopper is a free browser add-on that instantly compares prices and shows similar items on ANY product in hundreds of U.S. online stores including Amazon.com, Best Buy, Macys, Nordstorm, Overstock.com, Staples, Target, and Wal-mart."

So if I have this right, this is essentially a massive affiliate scheme to produce revenue for the company? If it compares prices on all these sites, affid='s are injected for Lenovo and a % of the sale is given to them?

Edit: doing the math here on this for the last few hours and even if just a few million units have been sold, this has to be 10's of millions in dollars (being very generous) over the past few quarters.

There reviews are horrible as well. All spam / annoyance related.

Given that antivirus products detect this as malware, does Lenovo not install any antivirus on their systems, or do they install a substandard one that fails to detect it?
Lenovo was the last respected PC laptop brand. Is there anyone I can trust to sell me a well-made laptop anymore besides Apple?
You can still wipe the hard drive and replace it with a Linux install (or a fresh Windows install, if you must).
Asus is about it.
I thought so too, but my recent experience with a Zenbook has changed my view. WiFi drivers were so bad it took half a year after my purchase before the connection became stable (not dropping every 15 minutes requiring a reboot). Touchpad drivers were also a mess with awful kinetic scrolling. And just couple of weeks ago it stopped booting Windows altogehter (something related to ACPI I guess, Linux works if I don't use suspend). Conveniently one month after the expiration of the warranty.
Despite the initial problems, I like my Zenbook.

The WiFi drivers are made by Intel, but yes, they were terrible (blue screen). I had to downgrade back to the drivers that came with Windows for while but the latest versions seem to be fine. I'm using some stock touchpad drivers that don't seem to have any kinetic scrolling.

But I'm the person who brought this to the attention of Hacker News: https://news.ycombinator.com/item?id=8546702

Basically after installing just about everything the laptop comes with, it seems to be running great. :)

Maybe Windows is terrible, I've heard lots of bad stuff about the wifi and touchpad drivers in particular. I installed xubuntu the day I got it and everything worked out of the box from day one. Power management on ubuntu 12.04 wasn't so great, but battery life became significantly better on 14.04.

Additionally, a roommate spilled a pint of beer on my computer and Asus replaced it for free, despite not having an accidental damage warranty.

I own two Asus products - Nexus 7 2012 and a K55VM series laptop. Both had problems with charging and used to get hot pretty soon.
Clevo makes high quality laptops that are also very reasonably priced when bought barebones from the right vendor (they don't do direct consumer sales). Sager, System76, FalconNW, and a whole bunch of other boutique laptop companies are actually selling rebadged/modified Clevos.
I have one of these. It feels very cheap and plastic. Definitely lacks the polish of a MacBook.
Jebus, how far the might IBM laptop line has fallen under the leadership of Lenovo. There was a time when a ThinkPad was arguably the best laptop money could buy. Many companies, including Google, would offer a choice between a ThinkPad or a MacBook, because those were the really reliable choices that were free of shovelware.

I even considered buying a Lenovo recently when a pretty nice looking ThinkPad was on sale, but a couple of friends have had very bad experiences with their Lenovo laptops. Both have had to go back to Lenovo for repairs; one of them had to send it back twice, and on the second go around demanded a new one instead of a repaired one, because the "repaired" one was worse than when it went in for repairs.

That said, there's "bad QC", which is forgivable with time and a sincere effort by the company to correct it, and then there's "evil". Intentionally shipping adware is evil.

Given this, I can genuinely think of no way for Lenovo to ever get my business for any product.

Is it even possible to buy a Windows laptop right now with only the OS installed?

This is exactly why I've been recommending Chromebooks to anyone who asks my advice for about a year now.

Microsoft sell their own laptops, in US. They are said to be good.
I think Microsoft sells those in its stores but even then I'm pretty sure they come with a few things but mostly from the manufacturer.

It would sure be nice to bring home a Windows machine that only had Windows on it and any necessary but minor applications from the manufacturer (like a settings application or drivers and not some photo sharing spyware).

You can buy "Microsoft Signature" machines from the MS stores and online. Hopefully the words will spread.
Wow haven't heard of those before, actually kind of like the idea of buying a PC and knowing there is an untouched version of Windows on it (unless you consider IE malware) :)
I bought my last laptop this way, and it's been very satisfying to own. There was no funny business, it's just straight-up Windows. It didn't even have any stickers on it except for a tiny Intel sticker.
MSFT should really be pushing these more, seems like a great opportunity
An unfucked machine is the superspecial case, something to boast about. Let that sink for a moment.
All laptops contain something which some people consider bloatware, because it is difficult to draw the line.

For instance, is it "only the OS installed" if it includes hardware-specific support for the display adapter, or a fingerprint reader?

Anyway, all laptops I have seen include either a generic Windows OS installation disk, or an option to order one for the price of mailing cost. But of course even with these you might have something included which you do not consider "only the OS".

That seems like a pretty easy line to draw. If the software is effectively a device driver - OK; otherwise - no.
Well, not for me. Like, what about the login management related to fingerprint reader? The reader and device driver are quite useless by themselves if you cannot use them for login. So the laptop vendor obviously bundles the driver and application together. And then you get an app that hooks itself in the place where you normally give your password. And might hook another application which does an alternative login method using the built-in camera (facial recognition).
With Windows even if you buy the boxed version it still doesn't mean you are free from hardware vendors fuckery. The necessary drivers are quite often bundled with shitware.
It's usually possible to unpack the driver installer, find the .INF file, and point Windows at it - this gives you the driver without any of the bloatware.

(An unnecessary hassle, I agree)

Yep. Especially with the fuckery that FTDI did.

What did they do? If they detect a "counterfeit" FTDI (in other words, a clone not necessarily claiming to be an FTDI), the driver bricks your chip!

Yeah, you can fix it using Linux, but it's a pain in the ass.

Or use Linux and be away from this cancer of MS Windows ecosystem.

Yes it is, you can even buy laptops with no OS pre-installed or a gnu/linux distro.

Chromebooks are the worst possible thing, I tell everyone to stay away from these crippled google branded piece of slavery.

I advise either a second hand quality laptop or a brand new one while budgeting a little extra for cleaning the crap that manufacturers preload inside to allow for such a low selling price.

Chromebooks are great. I've recommended them to at least a dozen people by now and they are all super happy with them. And free from MITM!
Free from Lenovo's MITM anyway.
Microsoft Surface is straight from MS - no bloat/malware. However I wouldn't buy it now since v4 is soon to come.
> Is it even possible to buy a Windows laptop right now with only the OS installed?

Microsoft's Windows Installation Media Creation Tool [1] enables you to download a clean Windows 8.1 ISO that can be used to re-install the operating system and wipe out all of the preloaded bloatware on any PC.

To do the same with a Windows 7 PC, visit Microsoft's Software Recovery website [2].

From Windows 8.1 Update 1 onwards, there is a built-in PowerShell cmdlet called Export-WindowsDriver [3] that will backup all of your third-party drivers prior to reinstalling the OS.

  Export-WindowsDriver –Online -Destination c:\DriverBackup

On older versions of Windows, DoubleDriver [4] is a good alternative.

Once you have created a bootable USB flash drive from the Windows ISO [5], another useful tip is to create a folder called $WinPEDriver$ in the root of the drive and copy the drivers you backed up into here. Windows will automatically install the drivers found in the $WinPEDriver$ folder during installation of the OS.

[1] http://windows.microsoft.com/en-us/windows-8/create-reset-re...

[2] http://www.microsoft.com/en-us/software-recovery

[3] https://technet.microsoft.com/en-us/library/dn614084.aspx

[4] http://www.softpedia.com/get/System/System-Info/Double-Drive...

[5] https://rufus.akeo.ie/

Good list of resources, but I'd like to add that the Windows 7 recovery page doesn't accept OEM license keys. If you try to enter the key from the sticker on your laptop, you will most likely be told to contact your hardware provider. Which means you're stuck with their crapware installer.
(comment deleted)
I've had great experiences with the ThinkPad T420, but after this news I'll likely never be buying a Lenovo product again. A damn shame.
I'm tempted to believe that's the last great Thinkpad. Until this morning I was being tempted by the new X1 Carbon, even with its non-traditional keyboard. Not so much now.
Not sure what you mean with "non-traditional keyboard", but Lenovo did change the keyboard in the 3rd generation Thinkpad X1 Carbons, reverting the layout of the 2nd generation to a more conventional one: with six rows instead of five. Glad they did.

Ars Technica just reviewed the 3rd generation version: http://arstechnica.com/gadgets/2015/02/thinkpad-x1-carbon-re....

As far as I am concerned this one has the non-traditional keyboard (CTRL is NOT in the lower left corner).

Mess with my muscle-memory and you're sure I will never buy your laptop. Same reason I'll never consider MacBooks: Non-standard keyboard.

Oh man I hate keyboards like that. If the keyboard is causing me to hit wrong keys, it's the keyboard that's wrong.
I used to feel the same until I remapped CapsLock to Insert on a MacBook running Linux so I could regain the ability to paste with Shift-Insert. After that I realized that none of my other keyboards had Insert in the same location, so having a non-standard keyboard wasn't unique to Apple. Now I try to remap certain keys on all my machines to the smallest set they share in common, so I can take my muscle memory with me.
In the BIOS for most Thinkpads I've used recently there is a setting to swap the Fn and Ctrl keys.
> As far as I am concerned this one has the non-traditional keyboard (CTRL is NOT in the lower left corner).

OK, that's one part of non-traditionalism :-) Luckily, the Ctrl and Fn keys' functions can be swapped in the BIOS (but obviously, the key labels will stay put).

I referred to the strange setup of the Caps Lock key, and the missing 6th row with function keys. (Although the functioning of the function keys is different in the 3rd generation model than in the 1st generation model).

My current T440s is pretty much all I ever wanted in a laptop. But yeah, this will make me think twice when the time comes to replace it. (hopefully not any time soon. Sweet sweet battery time!)

Then again, the first thing I did when I bought it was install an extra SSD and install Linux.

The T420 is, in my opinion, the last known good computer that Lenovo put out. I bought one in 2011 and still use it (sparingly) today. That is a rock solid laptop with a fantastic touchpad/keyboard.

We bought T440s a year or two later and both were just abysmal. The trackpad, the keyboard, everything is crappy and fails to work properly. No one at our company would use them and they sit in a closet now. I've been monitoring Lenovo's laptops recently and they all seem to be getting worse and worse.

You will find a lot of people who say things like: The [insert laptop model here] is, in my opinion, the last known good computer that [insert laptop brand here] put out. In the end it's just that, a personal opinion.

I have read similar things about basically every laptop(heck even cars, TVs, Fridges) brand in existence.

What was the point of this comment? I said in the first line it was my opinion.
FWIW, had pretty good experiences with the five Thinkpads, private and company boxes, that I was using at one point or another. There are things that could be (a lot) better - battery life on the W530 and, related to that, the ugly, ginormous brick of a charger that it comes with - but, all things considered, I will remain a Thinkpad customer, since I am not aware of better alternatives. The machines work without fail, and survive incidents like a fall from the overhead luggage compartment on a plane.

Crapware doesn't bother me, since that gets wiped before I start using the box, including the biggest offender of all them crapwares - MS Windows. Unless you're concerned about one of those disk-firmware-rewiring NSA uglies, that's a foolproof solution to the nastyware problem.

I have a X230 and I'm super happy with it. The quality is beyond everything I have experienced with laptops. I have a newer Dell E-series at work now and it's ok, but lack the same quality feel imo.

I suspect the cheaper Lenovo laptops are shitty though.

I can attest that thinkpad quality is on the decline, linux support too (not mentioning the stupidity of experimenting with new ways of doing keyboards[1]) but it's not that bad yet.

Hardware is good, in case of trouble on-site warranty works well (once you've learned your way through the ibm website). Be informed about what you buy, skip the comically broken models (see adaptive keyboard) use common sense and your thinkpad will be good. Nothing out of the usual when buying tech stuff.

Though in a not so distant future if lenovo declines continue, it may be wise to stay away from their brand altogether.

[1]: http://arstechnica.com/staff/2014/01/stop-trying-to-innovate...

Lenovo has learnt from that mistake though, the X1 Carbon Gen 3 basically has the keyboard from the Gen1 paired with the build quality and high quality IPS screen from the second gen.
This keyboard screwup was one of the reasons I went to Dell E series instead of Lenovo Thinkpad. If you are a heavy keyboard user, not providing dedicated Function keys is a big no-no. It is not about saving space either; my dell E7240 is only 12.5 inches but manages to have a fully functional keyboard. Besides, outstanding keyboard was a big part of the Thinkpad - what were they thinking mucking around with that?
I'll add I've witnessed bad mechanical design from Lenovo.

A friend bought a $1000 laptop (U330 touch) from them and a piece of plastic holding a hinge broke. When I looked at it, it was clear that the part could have been 10 times (yes, 10) thicker without adding much weight (about a gram I guess) and probably zero cost.

I find this mistake nearly unacceptable but the evil part comes when you ask for warranty and they tell you that you must have done something wrong, why would a hinge break otherwise? And you accepted the warranty terms, so its their right to say so.

Quality control also was an issue as the laptop first came with a malfunctioning keyboard and a non operating touch screen.

So yeah, now is not a good time to buy anything from Lenovo.

I'm planning to buy a new laptop in the near future and Lenovo definitiely goes out of the list. It's ridiculous where things are going in tech - everyone is trying to squeeze you like a lemon. Smart TVs that insert ads in your private videos and listen to everything you say, smartphones tracking your every move, e-mail clients scanning your mails, laptops installing spyware, cars that can be shutdown remotely, planned obsolescence getting worse and worse.. and that's only the tip of the iceberg - I wonder how much more similar bullshit is out there that we don't know about. Fuck all of that, I'll stick to good ol' "dumb" things as long as I can.
Exactly. But you also stated the reason - "everyone is trying to squeeze you like a lemon". Welcome to capitalism. At first, as the low-hanging fruits are collected, people benefit. Then, as Orz say, there is juice squeezing and then we are not so frumple.
I've purchased two post-acquisition Lenovos. A Thinkpad X1 Carbon first gen and, when it was stolen, a second gen. Both are truly excellent laptops, perfectly on par with the Thinkpad R40 and the X61t I had before.

The second gen X1 Carbon has two "innovations" I could live without. A clickpad and an LCD serving as function row keys. I must not be alone in my woes, as the third gen X1 Carbon reverted the change and has normal trackpad buttons and real function keys.

Other than that, the same quality Thinkpad build. It's not a war tank as the R40 was but, then again, it does not have the weight constraints that allow for a rollcage.

I know it is fashionable to say Lenovo fumbled the Thinkpad brand but, at least in the top of the line products, this isn't true. Of course, this is anecdotal, based on my company's purchases and nothing else. If you listen in forums, the landscape is much as the one here on HN (even if 90% of those who speak never bought a "chinese" Thinkpad)

Hell, who do we go with now? I'm a sys/web admin/devops by day and we just buy whatever is the hottest Lenovo, image them, and send them off for staff to use. They're rock solid from a hardware perspective and their laptops are usually top notch (ignoring the redesigned trackpad issues, they're pretty much perfect for business use).

We've tried HP and Dell in the past with the same ugly results. Horrible default images full of crapware, though not MITM bad. The only difference is that we had 10x the hardware issues with Dell and HP. We always need to make our own images. Windows OEM is a nightmare of shit crapware, which is a shame as the stock windows product is actually, dare I say, good? At least good for business use cases.

I also find it amusing that anytime there's some kind of issue in the US people instantly yell NSA, but thus far no one has thought to think this could be the CCP's attempt to spy on people by weakening SSL. I'm sure its trivial for them to grab the private key from Lenovo. Seems like the cyberwars are heating up.

Personally, I hope this becomes a major scandal. This deserves lots more press. In fact, every anti-virus product should remove this and the certificate. Anything short of that is irresponsible. This is congressional investigation worthy right here.

Oh, let us not forget the crap PC cleaner program that gets included in the Superfish install ("a Microsoft Partner"):

http://i.imgur.com/7cFlZLr.png

http://i.imgur.com/R4sHowP.png

So on a fresh Windows 7 virtual machine with zero apps installed, this program gives me 200 some errors and wants $49.99 (-$20 for instant savings) to register the program. This keeps getting better. Typical scam.