665 comments

[ 2.6 ms ] story [ 566 ms ] thread
Wow. Just switch per advice of lihkg! How serious? Seems all tech talk but not impact talk
> How serious?

From my understanding, that really depends what you're using it for. My friends mostly use Nord to get around region locks for Netflix etc. I think impact for them is minimal.

If you were using NordVPN in Hong Kong, to cover your involvement in the protests, then it could be a lot more serious. I wouldn't use Nord (or any comparable provider) for that anyway, since their holdings tend to be pretty opaque. That doesn't mean nobody did use it for stuff like that though.

I see the Twitter thread mentions:

> And someone just mentioned to me that past encrypted sessions may be able to be decrypted, which is a much bigger issue!...I haven't researched enough about OpenVPN to know if it's using forward secrecy, though you'd hope so

Any idea where that claim is coming from? Nord's site mentions having forward secrecy in place, so presumably most historical stuff is safe unless they botched that. Of course, somebody in e.g. Hong Kong could still have gotten a MitM attack if they were active while these keys were being used, which is reason enough to worry about exposure.

NordVPN is being recommended a lot to people who don't know better by influencers on social media, especially on YouTube. This kind of endorsement is recklessly negligent and needs to stop.

https://drewdevault.com/2019/04/19/Your-VPN-is-a-serious-cho...

Edit: note that I don't blame these influencers for their ignorance on the risks of using a VPN; rather I blame the shady VPN providers for overselling the security value of their product and leading users into a false sense of security.

I'm pretty sure they're "recommending" it because they're getting paid for it - it's a sponsor segment. After demonetization became common YouTubers looked for other sources of revenue and there are rather few companies that try to contact them directly for ads, so you see them appear over and over again.
Yeah it's diffidently not being recommended, it is being advertised. I wonder how many money they have spend. Every freaking channel mention them at some point.
You start to wonder where their money is coming from - their retail prices are already cheap, the discounts the influencers offer make it basically free. How's that sustainable?
No idea.

Yesterday I saw a discount with an extremely cheap 3-year plan (under 30$ and no data limit, iirc). The price didn't offer confidence that the service would be available for all three years.

But do they even have to pay much to youtubers for those ads? If you get 50k to 100k views per video then you'll likely make around the range of $50-$150 for the video. Paying the youtuber $50-$100 per video would already have a significant impact on their income, so they'd probably consider it. That would be 50k-100k people who will see the ad, because adblock can't block it.
If somebody is getting $50-$150 per video, they're probably doing it for the passion of making videos, not for the income, and they probably have another source of income that dwarfs what they're getting from youtube.
Not necessarily. If they put up a video every day then that's $1500 a month minimum. That's decent income in most countries, even in many EU ones. Now imagine if sponsor segments doubled that for you - now it's $3000 a month, which is already on the lower end of decent even in the richest countries.
I have a non technical friend who did this citing CNET. It felt like kind of a shit thing based on how it was being advertised, but I couldn’t actually see anything that warranted saying nord was bad.

How would have expressed this in laymen terms (before this compromised thing was revealed obviously)?

My layman explanation is:

You have to take your choice of VPN seriously. When you use a VPN, they can read all of your internet traffic, so choose a company you can trust with that information. If they screw up, like NordVPN did, then anyone can read all of your internet traffic even when you think you're safe. You're often better off without a VPN than with one.

This seems like an overstatement. Five years ago, mostly true, but can they mitm my ssl connections? (I'm getting mixed answers on StackExchange, but it seems like generally no.)

They can see what sites I visit, but for most of those sites, they still shouldn't be able to see the content.

(This might be more nuanced than the layman explanation needs to be. Just curious for my own sake.)

The article I linked in my original comment goes into a bit more detail and is aimed at the layman, but it's a bit more in depth than a comment you can make in a conversation.
It's likely that they cannot trivially MITM SSL connections but for that to be true you're relying on a bunch of things which are not trivial to verify:

1. All of the apps and sites you care about are HTTPS-only and don't rely on, say, an HTTP-to-HTTPS redirect which can be bypassed.

2. The VPN client doesn't do something like configure a proxy.

3. Your OS, apps, and browser don't have exploitable bugs or weak software update mechanisms, or that the VPN provider or whoever compromised them isn't going to try exploiting them.

Obviously the third one is a relatively low probability since it's noisy but it's the kind of thing which would be hard to rule out since VPN providers have a market incentive to cut corners if they think it won't be noticed and by their nature it's easy to imagine a law-enforcement or intelligence agency thinking it'd be a good service to compromise to get access to a userbase which contains people who are trying to hide something of interest.

Depends, if they have a root (or a wildcard) certificate, they can show you that, and your browser will happily show you a green lock. However, the list of root CAs in your browser is public, for Firefox see [0], and hopefully someone would notice if a VPN provider has access to such an certificate.

(However, that is something that also applies to ISPs, at least Telekom has a CA and therefore a root certificate.)

[0] https://www.mozilla.org/en-US/about/governance/policies/secu...

I also don't blame them, and in fact I'm a bit bullish on the fact that these influencers are bringing greater awareness to using VPNs to an audience that might otherwise not use them / understand them.
Snake oil salesmen have been around for centuries. When you have an audience of hundreds of thousands or even millions of viewers it's your moral responsibility to not betray their trust by recommending them bullshit. Unless you personally evaluated the claims of the product (definitely not the case as most of these people don't understand how a VPN works beyond "it somehow protects your privacy") and are happy to stand behind them, don't say anything.

In my opinion there's also another problem that needs to be considered, regardless of security skills: none of these VPN providers' business models are sustainable; they offer "lifetime" plans for cheap to begin with but also tack on extreme discounts (I once saw 83% off) in addition to paying influencers money to promote those discounts. There has to be a catch.

> none of these VPN providers' business models are sustainable

You can fit like a hundred of VPN users into a single cheap VPS server. With current prices for VPN they are anything but unsustainable.

Someone is probably going to ask what other HN users recommend as an alternative. Personally, I use Private Internet Access because they're the only provider I've found with a track record of demonstrably not being able to turn your records over to someone asking for them [1].

[1] https://torrentfreak.com/private-internet-access-no-logging-...

Yeah, they have been around for a very long time too. Used to be the more expensive alternative though, glad to see their prices have fallen a bit.
EDIT: I said I used IPVanish mostly because the EFF endorsed them, but someone pointed out below they got caught logging. I suppose that would explain why they're not endorsed on the EFF VPN page anymore. So, I guess time to find a new VPN. :(
Used IPVanish for a few years, it was great the first, sucked the 2nd. I switched. These days I use VyprVPN. I like their Chameleon encryption that hides the VPN (although some data still leaks and some sites still detect it) and the killswitch option (prevents connections out if VPN not active)
The thing is, almost all of these providers share infrastructure and IP blocks. Lookup MicFo and the associated lawsuits (not even including their lawsuit with arin). They provided the exact same IP blocked systems to dozens of the top VPN providers.

The reality is, if someone else owns the infrastructure you're just pushing the risk to a different location.

This is only an issue if the keys of the VPN are compromised (like it appears to be for NordVPN).
ProtonVPN comes from the same people who run ProtonMail, a very well known security focused email provider.
ProtonVPN has a large history of being connected to TesoNet, a company providing among other things data mining(!). An extra cherry on top of that is the CEO of TesoNet also being the CEO of CloudVPN, which more or less controls NordVPN.

Now that doesn't mean ProtonVPN is automatically compromised but I feel with stuff like no-log VPNs one should always err on the side of caution.

This has been thoroughly debunked, most recently by Mozilla and the European Commission as part of their due diligence.

ProtonVPN is 100% owned by the company behind ProtonMail, which in turn is funded by the European Union, so this has been verified by the European Commission. Details here: https://bit.ly/35RDKzB

I mean this[1] is pretty convincing and not directly from the accused company's blog. The only thing it gets wrong is framing ProtonVPN Lithuania as the main ProtonVPN company instead of as a subsidiary.

Regardless of that, there is so much mud being slung I recommend anyone to just search for 'protonvpn nordvpn tesonet', read a few articles on the topic and form your own opinion. Like I said, you can decide if you want to err on the side of caution or if it's a risk you're willing to take.

In case anyone wants VPN recommendations, I have good experiences with TorGuard and Private Internet Access and can also recommend Mullvad. Other people (that I trust) say iVPN and Tunnelbear are also solid.

[1] https://vpnscam.com/nordvpn-protonvpn-proton-mail-owned-by-t...

There's a couple ways to look at this.

On one hand, there's anonymous websites, competing VPN companies, and hundreds of Twitter bots pushing a story that is demonstratively false (just check public records).

Then, on the other hand, you have Mozilla and the EU (which has access to all European corporate records) vouching for Proton, which also operates in a highly transparent way, examples here: https://protonvpn.com/blog/is-protonvpn-trustworthy/

Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. And that doesn't mean Proton on a whole is based in Vilnius.

> On one hand, there's anonymous websites, competing VPN companies, and hundreds of Twitter bots pushing a story that is demonstratively false (just check public records).

I agree, the VPN industry is rife with shady business practices. But the story being pushed isn't 'demonstratively false'.

* TesoNet offers data mining services

* You did contract TesoNet employees

* Due to an error and unyielding policies by Google TesoNet holds your Android app signing keys in name

* There is a lot of intermingling between TesoNet and NordVPN and to a lesser extent TesoNet and ProtonVPN.

Like I already stated, it's very unlikely you are compromised. But unlike, say, a billing company that handles my energy or water provider (where I care much less if they have tenuous links to data mining) my standard is extremely high for a VPN. Internet traffic is supremely personal and for me to trust a company handling that there cannot even be the slightest sheen of misconduct.

For me to trust you you would have to completely cut out your Lithuanian subsidiary and any employees, board members, etc. that were or are related to TesoNet, as well as any reliance on their infrastructure. Obviously businesses don't operate with such 'scorched earth' policies and I don't expect you to gut your company based on a HN comment, but it is what it would take for me and many other privacy-conscious individuals to regain our trust.

Definitely appreciate your concern here, but there's still a lot which is being confused.

Proton does not today, and has never, used contracted (outsourced) employees. As is common with startups, in the past we did not always do all our HR in house (it's all in house today), but employees were always working on Proton and for Proton.

There are no board members, directors, shareholders, or employees, related to Tesonet beyond the fact that a couple employees might have been employed there previously. This in itself is not strange, we also have some employees who previously worked at Google, the ultimate data mining company, but clearly decided they preferred to work for the other side. People can and do change jobs.

Proton has also always run our own infrastructure, and for ProtonVPN, this is publicly verifiable.

So, we don't have to "gut our company" to remove any "intermingling" because there was little to none to begin with, and certainly nothing today.

Indeed trust is super important, but it seems odd to trust anonymous internet accusers or those with a clearly vested interest in harming Proton, as opposed to reputable third parties like the EU or Mozilla who don't have a vested interest here and are independent.

Proton is still to this day, the only VPN company that has an address clearly published on our website, where you can show up, and find company management and board members, and that means something.

Slightly off-topic but I am delighted by the generally non-abrasive way this thread is going. Dialogue is good!

I realized another way that would work for you guys (but is out of your hands) is fighting a court case about this. You'd be legally compelled to tell the truth and very screwed if you deny but then it comes out there is logging or mining going on. It's not ironclad but it is how most VPNs end up being considered 'solid'.

We have indeed retained lawyers to look into our options to fight the online defamation, but its hard to take anonymous accusers to court. However, as we have discussed here (https://protonvpn.com/blog/is-protonvpn-trustworthy/) there is already a lot of ironclad legal evidence.

First, were we to lie in our privacy policy, we would be subject to GDPR fines of up to 20 million Euros, since we have both European customers, and a presence in the EU.

Second, there has already been a court case. We were ordered by a Swiss court to hand over logs, and we stated truthfully (under penalty of perjury) that we did not have the logs requested. This case was previously disclosed here: https://protonvpn.com/blog/transparency-report/

'January 2019 – A data request from a foreign country was approved by the Swiss court system. However, as we do not have any customer IP information, we could not provide the requested information and this was explained to the requesting party.'

I'm not terribly well-versed in the international (or Swiss) legal system but are portions of that request public record, or would it be possible to put portions of it online, verbatim?

It would really strengthen the case to your customers because whilst claiming you had a request when you didn't isn't illegal, falsifying court documents definitely is.

No public indictment was issued because in this case the accused could not be charged since they couldn't be identified. Generally there are only documents if police decide to move forward with a prosecution, which is unlikely since we do not have logs that can identify users.
Anyone can set up an anonymous website and make spurious accusations and/or take money to post glowing reviews. The VPN segment is full of shady tactics like this. Never trust any VPN review site.

Trust serious organizations such as Mozilla and the EFF.

Mozilla trusts ProtonVPN enough to officially partner with them. That means a lot more than some random anonymous reviews.

> ProtonMail, which in turn is funded by the European Union

Wait... that doesn’t sound ideal either.

Over the course of the disclosure of the connection between NordVPN, Tesonet, and possibly ProtonVPN, Proton's story kept changing. They said contradicting things multiple times. They locked the Reddit thread. Why did Proton keep changing their story if they had nothing to hide? I will keep reminding this every time the issue gets raised. There is a compilation [0] of changing Proton's responses and them successively admitting more and more things not in their favor. The compilation starts at the part called "Online accusations fly".

[0] https://restoreprivacy.com/lawsuit-names-nordvpn-tesonet/

Both Mozilla and the European Commission have looked into the accusations being made on anonymous websites, and determined that they are false. The EU in particular, has access to records which allow independent verification.

There is also an abundance of public record which demonstrates this is false. The bad faith of those spreading this information is also apparent from the hundreds of fake Twitter accounts used to spread the rumors.

If you are acting in good faith, then we ask that you also take a moment to verify your facts and discover the truth, much of which can be found here: https://protonvpn.com/blog/is-protonvpn-trustworthy/

Can you confirm the certificate for Proton never had Tesonet in it? The number of overlapping coincidences is just unreal.
There's actually a point by point write-up about this on Reddit: https://www.reddit.com/r/ProtonVPN/comments/8ww4h2/protonvpn...

There's a historical, almost accidental connection dating back to the infamous November 2015 DDoS against Proton, but zero connection today, and certainly not in the way it has been portrayed by people seeking to attack Proton.

You're being downvoted because people believe in propaganda being pushed by competitors, but ProtonVPN / ProtonMail are very good options. Plenty of links and reports by Mozilla in this thread will lend credence to that.
Aka the conspiracy theory that was ultimately found to be pushed by PIA, one of their direct competitors.

I've got enough HN internet points that a few downvotes will be fine. Thanks!

> Aka the conspiracy theory that was ultimately found to be pushed by PIA, one of their direct competitors.

Source?

I've heard this several times but nobody has ever been able to provide a source.

It came from a protonvpn founder, an obviously biased source: https://www.reddit.com/r/ProtonVPN/comments/8ww4h2/protonvpn...

but was conveniently never denied by PIA that I could ever find. If it was a lie, it would be easy for them (PIA) to prove under libel laws in the discovery phase of a trial. I'd argue if it was a lie from ProtonVPN, it would have been in PIA's best interests to clear their name. After all, PIA and ProtonVPN are a few of the only providers who've proven in courts they don't have logs of users. We know they're legit because they said so in court under penalty of perjury. Also, the European Commission has investigated these exact claims, and would have privileged access to a lot of the business documents, and found the claims without merit.

Me? Just a happy protonvpn user who finds the oft repeated shilling for PIA dull. If you really want to hate protonvpn, use PIA, or use someone else. Better, don't trust any of them! Setup algo on a digital ocean droplet of your own: https://github.com/trailofbits/algo

However, this is meant for running over an untrusted network, not for maintaining internet anonymity. Use Tor for that.

I am surprised why isn’t anyone suggesting Cloudflare’s Warp VPN? Genuinely curious what is the difference. I guess Clodflare one is only for mobile?
Cloudflare's Warp is not an anonymising VPN as far as I know. It is just a way to speed up Internet speeds, especially in poorly connected areas. They make no effort to hide the origin IP. So it is not in the same class as other VPN providers.
This is interesting to read that Cloudfare is suggested here... shows that the term VPN is still thought of as private. (I know it is Virtual Private Network - but the termination is almost never private).
It's anonymous wrt your ISP. Which is what 95% of the complaints here refer to.
(comment deleted)
> I guess Clodflare one is only for mobile?

Yes, unfortunately it's currently not possible to use Warp VPN on PC. Otherwise, quite good service.

https://airvpn.org/ is also worth mentioning.

- Doesn't work for Georestriction - Shows your IP address (some pages) - You cannot choose your data center - Only for mobile
(comment deleted)
The claims there have been thoroughly debunked, most recently by Mozilla and the European Commission as part of their due diligence, details here: https://bit.ly/35RDKzB
Their Google cert literally is "Tesonet" - how is this claim debunked - you can check it yourself.
There's actually a point by point write-up about this on Reddit: https://www.reddit.com/r/ProtonVPN/comments/8ww4h2/protonvpn...

There's a historical, almost accidental connection dating back to the infamous November 2015 DDoS against Proton, but zero connection today, and certainly not in the way it has been portrayed by people seeking to attack Proton. Android certs are permanent and can never be changed so that is why there is still one mistakenly issued Android cert out there today.

For a company based in Switzerland to be "accidentally" connected to a company in another country they claim to have no connection to such that their permanent google cert lists the name of the company they are supposedly not connected with - that doesn't seem odd to you?

That fact that this had to be slowly pried out with changing explanations along the way?

When you say the claim that has been debunked - I expect the claim not to be confirmed.

Given all that is going on with VPNs, your caution is warranted, but one should also critically examine the claims that are being made.

Proton definitely has an office and subsidiary in Vilnius, it's not a secret because it's on our Instagram: https://www.instagram.com/p/BxMz62oHb6K/ The office is inside a 30 storey building, so it is not surprising the address is shared with quite a few other companies. That doesn't mean Proton as a whole is headquartered there, or that the subsidiary somehow controls the parent company in Switzerland, or that there is somehow data mining going on.

Those are the claims that have been clearly debunked. The fact that Proton has a subsidiary in Vilnius, or the fact that we outsourced our HR back in 2016, are not secrets, and is on our Instagram and the Reddit thread linked above. This is the truth, and this is not some wild EU-funded data mining conspiracy as some would have you believe.

Protonvpn is avoided by people who care about their privacy worse than a STD.
It's against almost all certificate standards for the certificate holder - Tesonet to allow an unrelated third party to use a certificate with their name on it. Or for a third party (proton) to use a certificate that has another companies name on it.

If you would like, I can pursue this issue further given what seems to be confirmation here of a certificate violation.

Apparently The Wirecutter now recommends TunnelBear and Mullvad because they post regular transparency reports and do third-party audits. https://thewirecutter.com/reviews/best-vpn-service/
I've been using Mullvad for a while now and I have nothing but praise for them. Only complaint is they're more expensive than some of their competitors.
What aspects of a VPN provider would you praise? Customer service, consistency of connection speed? Seems almost like a utility where it's hard to differentiate.
I am not OP, but they helped me with a rare openvpn config for which my forays into the openVPN forums led nowhere.

Out of my 110/12mbit connection Mullvad let's me use 108/11 of that, and even has wireguard support.

I would say transparency/security, quality of the client(s) and customer service, in that order. It's one thing to offer a VPN service, but to make sure you have a nice app on both iOS, Android, MacOS, Windows and Linux seems like quite an investment.
I have a slightly dissenting answer to these questions, in the form of an interactive Q&A website:

https://faq.dhol.es/@Soatok/cryptography/which-vpn-service-w...

How is this not the top comment?

Nobody should be using a VPN provider, full-stop. It is structurally impossible for anyone to verify their claims, they have more incentive to lie than your ISP does, and they're cheap and easy to set up, so the industry is a cesspool.

You should assume that all of them are behaving badly.

Plenty of people using VPN for: pirating, get an IP from a foreign country to bypass some content limitation etc., so they still need one, secure or not.
You can always setup Algo on your own personal VPS running on Azure, GCE, DigitalOcean, Vultr, etc.

https://github.com/trailofbits/algo

Also, hasn't CloudFlare been audited to verify their no log claims? I know while Mullvad hasn't been audited to verify their no log claims, they have at least been audited to verify the security of their app.

https://blog.cloudflare.com/1111-warp-better-vpn/

"1. We don't write user-identifiable log data to disk;

2. We will never sell your browsing data or use it in any way to target you with advertising data;

3. Don’t need to provide any personal information — not your name, phone number, or email address — in order to use the 1.1.1.1 App with Warp; and

4. We will regularly hire outside auditors to ensure we're living up to these promises."

https://mullvad.net/en/blog/2018/9/24/read-results-security-...

> they have more incentive to lie than your ISP does

A lot of ISPs openly collect user data, so I don't know how much that factor matters. And while I can go get a VPN, I can't just get another ISP.

Shouldn't matter to your average torrent user or someone who wants to watch US YouTube. And I guess that's most people who actually use it.
... this site is awful. It doesn't address the actual reason why people use VPN's. They don't want all their activities to be recorded/tracked by their ISP's (which depending on jurisdiction log everything for at least 6 months if not more) or other actors. And if somebody wants to deanonymize your traffic, they have to go to extra effort, whether it's by exploiting or establishing a relationship with your VPN host or whatever else. Or there are other use cases, like wanting to torrent in a country that is very liberal with serving fines (Germany).

And frankly, his alternatives are just absurd. Tor? Really? Has he ever tried to use Tor for usual daily browsing? Does he expect people to try to use Facebook, Instagram, Youtube over Tor? Really?

I've been using Private Internet Access (PIA) since 2016 and can also recommend it from a usability point of view. I'm not a security expert so I defer to others on PIA's security.
In 2015ish PIA got hacked via https://old-support.privateinternetaccess.com because of https://classichelp.kayako.com/hc/en-us/articles/36000646089... and never told anyone.

This bug loudly announces itself on every pageload, it speaks of tremendous incompetence that they ever let this go into production.

The site used to set a cookie that looked like this:

  Set-Cookie: SWIFT_client=a%3A1%3A%7Bs%3A15%3A%22templategroupid%22%3Bs%3A1%3A%221%22%3B%7D; expires=Wed, 28-Dec-2016 23:24:13 GMT; path=/; httponly
Obvious PHP object injection vulnerability that should've been caught by any automated auditing tool.
While the helpdesk software PIA used to use years ago did have that potential vulnerability, fortunately, Private Internet Access never exposed the support desk via plain http, and therefore, PIA itself did not have the vulnerability in its helpdesk.
Hahahaha, this bug was perfectly exploitable via TLS wrapped HTTP (so HTTPS, which is still HTTP as far as the PHP application is concerned).

The SWIFT_client cookie gets passed directly into unserialize(), TLS has literally nothing to do with this.

FWIW rasengan is one of the PIA founders, he should know much better.

This response is so utterly silly I must wonder if this is all just an incredible display of incompetence instead of malice.

Sorry, I glanced at the link you pasted and wrote the response as I knew this was a non issue from the past.

So, I spoke with our internal team and was able to find more details:

- We haven't used that machine since that exploit was made public.

- We were never exploited.

- There was no sign of intrusion of any kind.

- The specific machine was a backup helpdesk test server without any real user data.

Thanks again for bringing this up!

>- We haven't used that machine since that exploit was made public.

So what? You were exploited before kayako patched this bug, it was glaringly obvious to anyone who ever looked at the cookies set by your site.

>- We were never exploited.

This simply isn't true, either you're misinformed or lying.

>- The specific machine was a backup helpdesk test server without any real user data.

The specific machine (Which you took down really fast after I pointed it out! :P) I linked probably did not even exist in 2015, I was talking about your prod env.

I don't have a horse in this race, there's no incentive for me to lie about this. I know what you are saying isn't true.

If we're offering recommendations, then I'll go ahead and recommend Mullvad. They've got great clients for most common operating systems, good customer support, good performance, lots of servers to choose from, the ability to open ports, etc.

Something I find pretty neat about them from a technical standpoint is their account creation, user authentication, and payment processes. Sign-up literally takes less than a second, so even if you don't plan on using their service, I recommend you try creating an account.

I've had fantastic experience with airvpn. They're cheap, fast, reliable, and support all the configuration types you could want. I'm not affiliated with them but I'm surprised nobody here has mentioned them yet. By far the best VPN provider IMO.
Another vote for AirVPN. Good track record, happy customer for 6+ years.
PIA is also compromised. They installed the known criminal Mark Karpelès as CTO.
(comment deleted)
ExpressVPN - gets the job done, a bit expensive but rock solid reliable, breaches great firewall of china.
Freedome VPN has been similarly been shown to do the same.
Don't use Tunnelbear, they're known compromised. Honestly it's hard to beat Mullvad right now but that does make them a hot target, so keep your eyes peeled and know when to jump ship.
Any provider that offers that many IP addresses? I found NordVPN to be the only reliable service if you need to run requests from many IPs from different countries (web scraping).
I like Mullvad
Mullvad is a decent option, they have openvpn based app and wireguard profiles. Didn’t have any connectivity issues either.
I’ll second this. They also allow you to pay in cash which I think is an aspect of privacy that a lot of VPNs overlook.
It's odd that NordVPN, VikingVPN and Torguard all got their private keys leaked here.

- Did the hackers use an SSH or a VPN service vulnerability?

- Or maybe even a previously unknown vulnerability?

- Was SSH access firewalled? If not, why?

- Do they still have root access?

A lot of them shared physical infrastructure definitely less odd that multiple were breached at same time.
I was just thinking yesterday that people might be overly paranoid about that, I’ve always agreed that if security were important you shouldn’t share space but lately I’ve begun to question it since a lot of these data centers are pretty carefully controlled.

I’m glad I didn’t speak my mind on that I guess since I was wrong.

>It's odd that NordVPN, VikingVPN and Torguard all got their private keys leaked here.

Good reminder to set up FDE and not give your host logins for your servers. Unexpected reboots are rare enough that they're worth switching hosts over.

For dedicated servers this would work, especially for VPN where data-loss is "acceptable".

But if it where based on containers like LXC or OpenVZ, then the host can force root access via a command without even changing the root password of the container.

I doubt serious VPN provider are using LXC/OpenVZ containers. They don't even work with OpenVPN without special setup from provider, I don't know how about other protocols.
Both LXC and OpenVZ can run VPN services with just setting a couple of flags, I've done it on both.
I know, but these flags have to be set by provider of container, not user (provider of VPN). And they might require changes in kernel.
NordVPN _was_ (and certainly still is) using LXC containers. Look at the document linked in the Techcrunch article.
>For dedicated servers this would work, especially for VPN where data-loss is "acceptable".

FWIW there's no need for data loss when you ditch the server, just download the encrypted data and decrypt using a clean environment elsewhere.

>But if it where based on containers like LXC or OpenVZ, then the host can force root access via a command without even changing the root password of the container.

You should never do this, unless you truly don't give a shit about whatever you have on the server.

I mean that encryption puts the entire data-store at risk, I've seen it happen more than twice due to RAM being faulty (In one incident it was using ECC RAM) and a power-failure.

Even the backups where corrupt due to being backed up in encrypted images. When encrypted volumes and images are corrupted by RAM or power-failure, they are locked forever.

Of course one should never force root access, I'm saying that you can't keep out the hosting from access the server in that case.

> Even the backups where corrupt due to being backed up in encrypted images. When encrypted volumes and images are corrupted by RAM or power-failure, they are locked forever.

That sounds more like issue with backup procedure (and testing of backups), even if it was amplified by encryption.

> Of course one should never force root access, I'm saying that you can't keep out the hosting from access the server in that case.

LXC and especially OpenVZ containers seems to be replaced by KVM in hosting/cloud. Of course, it's still possible to attack VM as host has control over VM's memory. Even dedicated servers are potentially vulnerable to attacks like cold boot.

> In one incident it was using ECC RAM

Did it at least warn about issues or was it ignored?

> I mean that encryption puts the entire data-store at risk, I've seen it happen more than twice due to RAM being faulty (In one incident it was using ECC RAM) and a power-failure.

How can this cause data loss? Header containing encryption key should not change during normal work. Did it just corrupt writes?

Yeah this sounds so weird. Data loss due to bad RAM will be extremely similar with or without encryption, it's not going to corrupt the header of the disk.

The catastrophic data loss you describe almost certainly resulted from you doing something horribly wrong, and not encryption.

or it's the same company ;)
Answer: Compromised datacenter that had insecure remote management system running.
And only few data centers that want to have VPNs on their hardware (and offer cheap bandwidth).
A while ago I read that there was a potential smear war going on between some of the larger VPN providers. Is there any chance that this is related? (I'd prefer more than just a tweet)
If you care less about the pseudo-anonymous-but-not-really shared-IP aspect of using a VPN, and care more about the this-lan-is-sketchy use case, I have had good experiences with Algo [0]. You can just paste in an API key and spin up your own VPN on something like DigitalOcean. And it uses WireGuard!

[0] https://github.com/trailofbits/algo

I was getting ready to paste the same thing; the command-line instructions make Algo, effectively, a somewhat technical solution, but it really does just work.

And for those who have used various VPN solutions over the years but not Wireguard: it really is pretty magic. It Just Works, with fantastic performance.

There's also Streisand[0], that gives a lot more options.

[0] https://github.com/StreisandEffect/streisand

Came here to post this. I’ve been using streisand for a long time with no problems. I’ve given out logins to a few trusted friends / colleagues and all have had good experiences as far as I know.

Plus I really enjoyed learning about the in’s and outs of setting it up. I poke around in the VM just for giggles.

Ive done this but have found that most services (Netflix, etc) recognize DO as a VPN. Does anyone know of a hosting provider that isn't blacklisted but I can still setup wireguard on?
This. I can set up and connect to a new OpenVPN instance under my own control in less than seven minutes (5:59 last I clocked) from my phone.

Anyone can do this, it’s not nearly as complicated to launch and secure as some would have us believe. (https://github.com/jenh/sevenminutevpn)

You do lose anonymity with personal VPN, but it all depends on your use case.

But the problem remain the same. Whoever manages the network that hosts your instance will see your traffic...
Yes, it's not a perfect solution. You're basically deciding who you trust the least and avoiding just them.
You choose the business model to trust. VPN serve customers who to pay to be private. That seems like a high value target. ISPs serve connectivity, but apparently in the US, spying on their users is part of their core business and have strong local monopolies. Due to fierce competition and trust being pretty much one of the business requirements (I'd expect a lot more due diligence in b2b), hosting providers seem like the least big evil to me.
There are places that rent boxes for XMR if you want a hosting provider who doesn't have your information.
If you already have a DigitalOcean droplet up and running and you have ssh access, you can use sshuttle [0].

e.g. run this from the command line:

  sshuttle -r example.com 0/0 -x example.com --dns
[0] https://github.com/sshuttle/sshuttle
OpenSSH also includes a SOCKS proxy which you can use with no additional software: https://ma.ttias.be/socks-proxy-linux-ssh-bypass-content-fil...

Whether it grants you any significant anonymity is debatable, but it works well for evading content filters and tunneling your traffic onto a more trustworthy network.

Speaking from experience, sshuttle is way easier and more robust than using OpenSSH's built-in SOCKS proxy.
I use sshuttle all the time when working from "restricted" networks (car dealerships, airports, etc.) For some reason, my local Honda dealer has a guest WiFi that restricts outgoing traffic to a small number of ports, and apparently SSH isn't on that list, so I can't push/pull to GitHub. Firing up sshuttle on port 80 punches right through the filter and allows me to do real work while I wait for my oil change.
The problem with this is that jumping out onto the net from a VPS-allocated IP causes all sorts of trouble for "normal" internet use. For example you won't be able to use Netflix doing something like this.
I can see why Netflix would try to block it, but I haven't run into any issues with it myself (OpenIKEd on OpenBSD on a $3.50/mo Vultr server as detailed here: https://www.snazz.xyz/how-to/2019/09/13/vpn.html). A lot of websites seem aggressive towards Tor users, but my VPS IP address was treated the same as my home, work, and LTE addresses. Are there any other documented cases I should be aware of?
Vultr is a lesser-known VPS, ymmv. I had issues with several websites using Linode. I don't have a list, but iirc it was some gaming related service that took issue with the IP.
Thanks for this. Just got it up and running in about 15 minutes (most of that was waiting on DO setup and scripts to run).
Is Algo really 1 ip = 1 user? I always assumed multiple things could be running on one IP since IPv4 is getting scarce, but bare metal networking is not my expertise.

I use Algo for the exact reason you mention ("this lan is sketchy") and have been pleased, but I always assumed even if my traffic was mingling, one (possibly secret) court order would out me since I paid with a CC tied to my real name.

Another option is Outline VPN, Jigsaw/Google's open source implementation of Shadowsocks:

https://getoutline.org

https://github.com/Jigsaw-Code/outline-client

https://github.com/Jigsaw-Code/outline-server

Shadowsocks is more resistant to censorship from adverse actors (such as the Great Firewall) than OpenVPN.

Outline's user experience is the best I've seen among self-hosted VPN solutions, as it includes apps for both the server and the client. The server app is suitable for use in organizations, and can manage VPN profiles for multiple individuals.

It's an alternative for sure and has specific use cases, but calling Outline a VPN is disingenuous. It's just a Socks proxy with some obfuscation built in.
Shadowsocks handles all of the use cases of a VPN. When all of a device's internet traffic is routed through Shadowsocks, there is no functional difference to the user. This is the default behavior for all Outline clients (desktop and mobile).
Nothing on this page or on the trailofbits blog article tells me why I should actually use this. Why should I trust DigitalOcean more than <insert VPN provider here>? Especially when it says "Does not claim to provide anonymity or censorship avoidance" - why would I use a VPN if it can't even attempt to provide some measure of anonymity?
More worrisome is that this may allow old cptures to be decrypted, anyone know of VPN providers that use PFS?
Private Internet Access is PFS (using DHE - Diffie Hellman Ephemeral).
Nord (and perhaps others) seem to have been compromised for months/years - lifetime accounts have been available on the DN for significantly cheaper than other VPNs: https://news.ycombinator.com/item?id=20094946

Doesn't seem like a smear - glad this is coming to light.

Nord specifically has a retailer system, I wonder if accounts can be created 3rd party and sold greymarket like that without any nefarious hacking scandals?
It's more that their users got hacked.
I don't understand the obsession with VPN providers. Funneling all your Internet access through a single entity no matter where you connect from just seems like a fundamentally bad idea to me, especially if that entity's business is getting people to funnel all their traffic through, making them a juicy target for governments or hackers.
Because for certain types of things, like pirating movies, it's good enough.
It seems a bit strange to me that you'd want to hide your movie pirating from your ISP but are happy for your VPN provider to know about it.
I think it is more of a matter of hiding your local ip address from other torrent users.
What's the purpose of that? Genuine question, I'm trying to understand the threat model of another torrent user seeing the IP address that is currently allocated to your router. Is this related to those letters that people sometimes get sent threatening legal action for torrenting?
quite simply evading copyright strikes or legal filings. I don't know which providers still disconnect your internet after 3 strikes. But I am guessing that's the "threat" here.
Any (large?) ISP will report your torrenting and/or terminate your internet usage if you torrent anything they deem copyrightable. Both Comcast and Spectrum do this, at least.

Edit, to add: No VPNs do this.

OK, I was not aware of this. I have been known to occasionally torrent things (not just Linux distros!) and having done this semi-regularly while living in four different countries I'm yet to hear anything from any ISP about it, but I am sure there are places where it's taken more seriously.
> No VPNs do this.

Yet.

> No VPNs do this

I got a nastygram from the hosting provider I used.

They don't. What you're describing would take them out of common carrier status and make their business unworkable. What they do is respond to notices from copyright holders.

https://arstechnica.com/tech-policy/2011/07/major-isps-agree...

My friend's VPN emailed him saying Comcast asked them for his info because he torrented something and the copyright holder contacted Comcast. The VPN provider didn't give Comcast any info.
Your friend may need a new VPN company. If they were able to identify him then they are probably logging too much information.
Agreed - I was oversimplifying. A better way to put it would be to say that ISPs automatically participate in the reporting of alleged copyright infringement, as well as mete out punishment for alleged infringement when they decide it is appropriate, whereas (most?) VPNs do not.

Edit: spelling

Who said anything about being happy about a VPN provider knowing anything?

It's about choosing between the lesser of two evils.

I am guessing #1 is mot wanting your internet provider (eg. AT&T) knowing what you are doing, then Netflix, Torrents, getting better deals on tickets and such, maybe activities of questionable legality?

Personally, I don't like the idea of my mobile provider profiting off knowing which applications I am using and what sites I visit.

I don't understand being unhappy with your ISP knowing these things, but being fine with your VPN provider knowing them.
It is much easier to change VPN provider than your ISP.
What does that have to do with them knowing what websites you visit?
The implication is that if they're doing shady shit, you can easily switch, in contrast to ISPs.
On the other hand you will have lesser knowledge of what shady things your VPN provider does.
Because it's easy to change a VPN provider if you don't like their actions, but most of us are stuck with an ISP and have no control over what they do with our data?
What exactly can they be doing with your data other than selling a list of which DNS queries you make and which IP addresses you connect to? (Which the VPN provider can also do.)
Send me your browsing history and your location history and we can continue this conversation.
Where did location history come into this? (IP addresses are generally not correlated to location at much more than city level.)

My point is simply that using a VPN provider doesn't change the fact that an actor has access to your DNS queries and which IPs you connect to (and where you connect from). It just changes that actor from your ISP to a VPN provider, and most VPN providers seem a hell of a lot more shady than any ISP I've dealt with.

I'm currently sitting on a plane. gogo wifi can read all my email if they so chose (and pass to a government). I'll use a VPN.
Any major email provider at this point should have SSL enabled when you check your email.
The browser (as should your email client) appears to have rejected those certs.
ok, so they have been rejected. Now what? I still would like to read my email... which is where VPN's come in handy.
I have had this situation in-flight a few times. I just used wireguard to a server I have. If I didn't already have that set up, I would have used an SSH+SOCKS tunnel to route around the damage. No need to send all my traffic to some shady VPN provider.
If you're running your own VPN, fine.

If you're using something like NordVPN, you're just swapping one shady outfit for another. Worse, in many cases.

Might be better to just wait until you land.

FTC believes location data is being sold alongside browsing history.
The ISP knows who you are and where you live; the VPN provider only knows your source IP address and information gleaned from your payment method (which in many cases can be "not much", as VPN providers support pseudo-anonymous payments).
Many of us live in countries where our ISP is owned by a hostile government that regularly blocks social media and other parts of the web on a whim.
They can do active attacks on you, as most people don't actively attempt to ban and absolutely block unencrypted connections (and there are also sometimes attacks on SSL stacks anyway); and like... SSL isn't really designed to protect the content of your connection anyway: due to size and timing attacks, people have deployed practical implementations of stuff like "figure out where I am looking at on Google Maps" and "figure out what movie I am watching on Netflix", and while I haven't seen a practical implementation of it yet, "learn too much about my search queries due to find-as-you-type".

(Also, if I see you making requests to some websites I can correlate it to others, just on hostname, which I would get from SNI/TLS, not DNS: like, you go to news.ycombinator.com followed by some other websites that are currently on the front page of Hacker News, I can now guess with high likelihood you are clicking on specific website links you just saw.)

As for "the VPN provider can also do that", that is like saying "what can a random stranger do with your secrets that someone you know well can't?", which is "true" sure, but not really interesting: being able to choose the company on whom you rely for security is extremely useful: I don't really have choice over my ISP, but I have choice over my VPN, and so you can't really say "these VPNs are shadier than my ISP" unless you can show the best of all VPNs is shadier than my ISP.

Meanwhile, for many people, your "ISP" on a given day might be "the local coffee shop" or "an airport" or "your brother's friend Bob": people talk about "ISP" as if it always means "AT&T", but I see even extremely technical people who "should know better" happily using WiFi provided by conferences, which is just crazy to me... you are way more likely to get messed with in some scary way by people close enough to you for it to matter than by some random entity.

> SSL isn't really designed to protect the content of your connection anyway: due to size and timing attacks, people have deployed practical implementations of stuff like "figure out where I am looking at on Google Maps" and "figure out what movie I am watching on Netflix", and while I haven't seen a practical implementation of it yet, "learn too much about my search queries due to find-as-you-type".

A VPN won't protect you from these sidechannel attacks.

Not by default, but it could. Send a monolithic stream of 1500 byte packets with some padding to obfuscate transfer rates and you can really disrupt that kind of thing.
I can use a VPN provider outside the jurisdiction of my own country.
If you're doing something illegal in your own country, that seems like a good idea. If you're not, that would seem to achieve nothing other than making it much more difficult to enforce any action against the VPN provider for selling your private data.
If my VPN provider trades user data, the service will quickly deteriorate and it won't be a VPN provider for long. But even if that is the case it wouldn't be my primary concern.

I don't even live in a country were I have to fear much at all from malicious authorities, but they wouldn't even blink before trading privacy for perceived security.

I might change my opinion if there were actual consequences for sharing user data. I believe it when I see it.

Otherwise I just like privacy, information is power and I don't like to share with the state.

In the US at least, there still remain a few tatters of laws that control how law enforcement and intelligence agencies can surveil you.

But there are zero controls on US agencies hoovering up data indiscriminately outside US borders.

Well your ISP knows more about you than your VPN necessarily does. Your ISP probably has your credit card on file, with your real name, and they have your precise street address too. The VPN may have none of that, except your IP address. If somebody were to purchase your history from your VPN, they would have to also purchase the IP->name/address/etc mapping from your ISP and JOIN the two. That seems marginally better than a one stop shop.

(Of course, some people give their VPN their credit card info, so the above rationale doesn't apply for them.)

Because they're required to reply to subpoenas?
I feel like the only good reasons to use a VPN are if you're torrenting or if you want access to sites from different countries (foreign Netflix libraries, streams from state-run media channels, etc). Most VPNs worth a damn aren't going to sell you out just for torrenting movies/music/games while your ISP will.
I use one when I have to connect to a public access point, or really any network that's not owned by someone I know and trust. It's not a perfect privacy/security plan -- you're right that I can't completely trust the VPN provider either -- it's just better than the alternative.
Try Warp from Cloudflare. Does the same thing but easier to trust versus some outfit like NordVPN
Well, you're funneling your traffic through a single entity in almost all cases, right? So I view it more as, who do I distrust more? My ISP or a VPN?

I don't use a VPN provider, but it's tempting as I don't trust my ISP at all.

One very explicit reason to not trust your ISP with your internet traffic is that since 2017 [1], they are allowed by Congress to sell your internet history.

As a cherry on top, they were also the ones that successfully lobbied the government to allow that in the first place [2].

[1] https://www.privateinternetaccess.com/blog/2017/03/house-rep... [2] https://www.privateinternetaccess.com/blog/2017/02/internet-...

To be fair, they can't actually see more than hostnames & IP addresses (assuming the use of TLS, which is becoming ubiquitous), so implying that they sell your "Internet history" makes it sound worse than it is.

I've always assumed VPN providers sell whatever data they can too.

I’m not a network expert, but doesn’t TLS just cover your connection with a specific website? Since your IPS is often also your DNS, can’t they still see which specific websites you’re trying to connect to? Wouldn’t TLS just obfuscate what you’re specifically sending to and receiving from that site? I’m under the impression that my ISP can (and probably does) see every website I visit, which is in the least browsing history.

I also remember that Comcast did (and might still do) inject code onto websites to display a “pop-up” indicating that you’re reaching or have reached your datacap. It would even pop up on Steam because most of Steam is really just a webview. I’m not sure exactly how they did/do that.

Again, I’m not a network or security expert, so I’m not really sure of how TLS protects your internet history, which I take to mean a list of websites you visit and when.

I believe you are both saying the same thing, you called it websites/domains he called it hostnames.
That’s what the parent said: they can still see “hostnames and IP addresses.”

But, for most people, that means that the ISP will just see:

• google.com

• facebook.com

• reddit.com

• somebignewspaper.example.com

Etc.

And there’s really nothing much too valuable about that. They won’t even be able to figure out if you’re shopping for something (unlike every other nosy channel provider), because most shopping traffic today just looks like Google + Amazon.

I see. As someone who has practically zero knowledge of networking, I had misconstrued hostnames for something else, which I’m now too embarrassed to mention.

Very educational response though. Thank you.

Please don't underestimate the value of metadata.

While it's true that the big platforms dominate web use today, don't forget that the concept of metadata includes when you actively surf on the information superhighway. That's valuable information for advertisers.

So is every DNS lookup related to the API backends of specific apps you use. And every random website outside the massive platforms.

These might reveal tons of information about you. Like:

Are you doing research on politics (and which flavor)? Do you worry about health? When do you access online banking? Which banks? Any tax filing software? Invoicing apps? Do you use shitty payday loans? Are you looking for dates? Are you gay? Which games do you play? Which car dealerships do you consider? Do you gamble? And of course, any particularly.. specific porn sites? Do you access banking, travel/flight booking, investment, shitcoin trading, adult or gambling sites in a specific pattern that might indicate mania or other mental health issues?

With metadata alone, your ISP has a thick dossier on your habits, with stuff therapists don't know about their clients.

(comment deleted)
You're correct - TLS / SSL operate above the IP layer, so they know which servers you're talking to, when and approximately how much, and if they can see your DNS requests it makes it that much easier to know which sites that maps to. TLS stops them from seeing the actual data you're sending them (like passwords), which pages you're viewing, and it also prevents them from manipulating the data (unless they've subverted the PKI, like if your company laptop has a certificate installed that trusts your company VPN, you company VPN can do a Man-in-the-middle attack to subvert TLS).
By looking at side-channel data it was possible to correlate the page being viewed on SSL, IIRC researchers could calculate the page viewed about 70% of the time.

I misremembered, it was about 90% -- https://scirate.com/arxiv/1403.0297.

... and request sizes, relative times, and time of day. It would be foolish to not assume that the complete history of your sessions can be inferred from how this data clusters, everything but the actual text of your messages.

Of course the people that find this problem worthwhile to solve then go on to work for or found surveillance companies, rather than publishing proof of concepts to security lists.

We also already know the type of molds the surveillance companies are trying to fit us in, from their own marketing materials (eg https://www.experianintact.com/content/uk/documents/productS...). Do you really think there isn't enough metadata being leaked to bucket people into these categories?

And yeah, IP proxying is a hack. But it's seemingly the best we can do to mitigate the utterly broken HTTPS/JS protocol stack.

There are other straightforward advantages too, like having location targeting miss the mark which breaks up the coherency of their manipulation. I've got zero intrinsic interest in local/news events for elsewhere.

TLS has optional padding. In TLS 1.3 clever design means the padding is "free" (each byte of padding adds exactly one byte of data transmitted) so if you would like the sizes transmitted to be misleading you can choose how much.

We can't solve for you the question of how much to use. If you want a snooper to not know if you retrieved file A of 14583 bytes or file B of 14621 bytes maybe a very small amount of padding will get the job done. If file B was 800 Mb that's a lot more padding you're asking for.

Sure, but that doesn't really address how clients/websites use it right now, or even scale up to solving the fundamental problem (the best you can do is hide bits by padding requests/responses to a discrete set of lengths).

If you're responding to my characterization of HTTPS/JS as "broken", I'm referring to the fact it needs to make a connection to a well-known centralized-authority server every time it wants to retrieve a resource, leaving you at the mercy of your transit (and the server itself, which is obviously another major source of surveillance). Whereas something based on ideas like content-centric networking (eg Freenet) allows a user agent to retrieve those resources from peers or broadcasts, perhaps even over virtual constant-bitrate links.

If the NSA revelations have shown anything, it is that meta data is very valuable.
>Funneling all your Internet access through a single entity no matter where you connect from just seems like a fundamentally bad idea to me,

You're already doing that with DNS servers. At least VPN providers make the claim that they'll protect you.

Sigh - just because one has a VPN, doesn't mean it has to be used all the time.

My NAS device, uses a VPN - my other machines do not.

My laptop, uses a VPN when I am travelling and using wifi from unknown sources (i.e. coffee shops, airports, etc.)

"I don't understand the obsession with VPN providers. Funneling all your Internet access through a single entity no matter where you connect from just seems like a fundamentally bad idea to me, especially if that entity's business is getting people to funnel all their traffic through, making them a juicy target for governments or hackers."

Agreed - especially when it is so very, very cheap and easy to fire up a handful of VMs - around the world - and run your endpoints any way you like.

By far my preferred technology is 'sshuttle'[1] which allows you to use any host, anywhere, running ssh as a VPN endpoint. That cuts the setup time for your VMs down to almost zero.

[1] https://github.com/sshuttle

You can fire up VMs, but then issue becomes the VM provider, which definitely has their own logs. Doing torrents through a VPN you setup on a VM provider usually gets the automated emails being forwarded to you.
> Funneling all your Internet access through a single entity

You don't have to funnel it all, only data crossing hostile networks like free wifi hotspots (The only real use-case for VPNs in the first place). Alongside this is choice of geolocation so you can watch things like HBO even when in Europe.

Regardless of anonymity or privacy, my ISP almost certainly deprioritizes some traffic (video), so a VPN at least ensures I can use my full bandwidth for whatever I want to access on the Internet.
NordVPN just posted this a few minutes ago: https://nordvpn.com/blog/official-response-datacenter-breach...
I'm curious about who this cloud provider might be.
The local IP for the OpenVPN endpoint (185.212.149.9 listed in the gist) belongs to creanova.org since 2017.
They wrote:

> We […] started creating a process to move all of our servers to RAM, which is to be completed next year.

What does "RAM" mean here?

I guess that all decryption keys are on ram. If the power is disconnected, then it would need a manual intervention to re-decrypt the data
Likely booting the OS from a RAM-disk and mounting as read-only.
NordVPN is down, also looks like VikingVPN and TorGuard were affected as well: https://twitter.com/cryptostorm_is/status/118609795032747622...
The thing that scares me here is that these keys were leaked May 2018, and it's becoming public knowledge now.

Someone found certificates for those three VPN providers and posted them to 8chan with a message like "I don't recommend these VPN providers lol"

The good news is that they're only certificates, and they have now expired, but theoretically they could have been used for the past year without anyone noticing.

During the spate of health care information leakage, someone invented a MTBCA, meaning "Meantime to CEO Apology" for the time between the breach and the CEO apology. At that time, it was running on the order of 8 months.
I guess it depends what you want from your VPN.

When I want to secure a shady connection in a coffee house, I have a raspberry 3 at home that I use only for that purpose with an openVpn setup with https://www.pivpn.io/ - super easy to use. Downside, I rely on my isp not to spy on me. Upside, it's mine and unless I'm specifically targeted it's unlikely someone will mitm me.

To hide my location for various purposes, I have used TigerVPN. They have been reliable so far, but I wouldn't trust entirely any third party when it comes to privacy. Upside - somewhat reliable and not my isp. Downside - for all I know someone in Czech Republic is watching what I stream with a bucket of popcorn

Lots of talk here from highly technical folks but not one person brings up the fact that these are expired keys - as in not usable?

I understand that the fact that these keys were obtained is concerning but the security of nord and etc prevailed at the end of the day.

The question is: were they leaked before they expired or long after?

> However, the key couldn’t possibly have been used to decrypt the VPN traffic of any other server. On the same note, the only possible way to abuse website traffic was by performing a personalized and complicated MiTM attack to intercept a single connection that tried to access nordvpn.com.

However crt.sh shows

> Validity > Not Before: Oct 6 12:53:38 2015 GMT > Not After : Oct 6 12:53:38 2018 GMT

What exactly were these keys for if they were only usable in such a manner according to nord?

Nord has a couple thousand severs, and each has their own key. In order to decrypt traffic, you'd have to intercept some traffic to decrypt, which would require a MitM attack unless you're an ISP/state actor.
Depending on the web server configuration they could be used to decrypt past traffic.
I've just read the official statement and it seems that the data center was compromised, not the VPN traffic, it wasn't affected by this
This is troublesome.

I was planning to eke out $85/ annum and go for NordVPN, but now even this is unreliable

Im using PIA for 2 years now - pretty happy with the service.
Yeah been on PIA myself for 5 years. Amazing speeds and service. No issues at all and they have been in court twice where they showed they have no logs.
Its great - i use it on my work PC to reroute my personal traffic, on my families personal devices, when traveling abroad etc. That said, there is a no 100% guarantee so use it responsibly.
Except this isn't their fault because their infrastructure provider messed up and didn't even disclose this possible backdoor. If anyone the provider should be named and shamed, not NVPN.
> NordVPN said it found out about the breach a “few months ago,” but the spokesperson said the breach was not disclosed until today because the company wanted to be “100% sure that each component within our infrastructure is secure.”

Not acceptable.

Yes it is?

Giving realtime public status updates makes your attacker privvy to your actions and how much you know. Fix first, publicly announce when safe to do so.

Same reason why SWAT etc. don't want media crews covering their actions in real time. It's broadcasting your view of the situation and intent to the opponent.

What should they have done if they knew about a potential infrastructure weakpoint? How should they announce that to the public before fixing it?
If you're making a living selling a secure channel, where the whole point is to be more secure then other channels, you better fucking secure that channel. You can't outsource the underlying hardware and then wash your hands of what happens.
Users trust The VPN services expertise in managing all aspects of the network to be secure. Nord failed by not managing the provider relationship to the level of security they promise to their users in their marketing.

I'm not saying it could or could not have been prevented. It is a fact that they did NOT live up to their promise made to their client users.

There must be some independent certification of these VPN providers by some sort of industry association to restore confidence in this non-transperant product that promises protection.

After reading their blog post about it [ https://nordvpn.com/blog/official-response-datacenter-breach... ], I'm not as concerned.

A compromise is a compromise, don't get me wrong, but it can happen to absolutely anyone. If you're paying ~$5/month for anonymous browsing with unlimited bandwidth, then you're probably not getting top tier security researchers running your servers.

(comment deleted)
Buy a $5/month VPS and run your own VPN on that (popular setup script: https://github.com/StreisandEffect/streisand). It'll cost you a little bit of time in setup and maintenance (mostly just upgrading packages), but it has many benefits:

- Cheaper than most VPN providers

- You won't be using a known VPN IP

- VPN providers are more likely to snoop on your traffic or be targeted by snoopers (such as the government), specifically because they seek out traffic from people trying to hide

- You get to pick the port/protocol/software you use, rather than being forced to accept the provider's ones

- You can run other small servers you may need on the VPS as well

But then your security rests on your ability to manage a server. I mostly agree with you, but, I don't run one because I'm not a seasoned Ops. At least, not enough that I want to put my security on the line.

In all but the most hostile networks I trust another VPN or my ISP more than I trust my ability to keep a server secure.

Thoughts?

Some thoughts:

1. You have to keep two ports locked down. If you can secure your own laptop, you can secure a cloud instance. The cloud instance you're basically just using as a proxy is a lot less important than what's on your phone or computer.

2. Only you are using the system, and you're not logging. Have an issue? Tear it down and start another. Automated scripts out there generate unique keys every time.

3. A commercial VPN is a honeypot in a way -- it's a ripe target. Many people are tunneling through it, doing sketchy things that certain parties want to track -- and your traffic could get caught in a dragnet (this, of course, depends on your use case: you may want to blend in).

4. Your ISP tracks and sells your data. I mean, the entire reason I use a VPN is because I was sick of my ISP routing my searches through their servers before my intended search destination, snagging my Netflix info and using it to create advertising profiles. Why would you trust them?

5. It literally takes less than 10 minutes (5:59 from an iPhone, the last time I launched one) to launch and connect to your own VPN instance to play with (https://github.com/jenh/sevenminutevpn is mine, but there are others, like Streisand or Algo) -- if nothing else, you become a more educated consumer and can better understand your threat model AND what to look for in a paid provider.

> You have to keep two ports locked down

That's what NordVPN thought as well

I would hope that isn't what they thought and I'm sure it's not what they thought.

Launching a personal-use ephemeral cloud instance running OpenVPN to hide your personal traffic from your ISP is absolutely nowhere the same as running a paid VPN service for millions of users across the world.

Using this for scary public WiFi makes some sense. Beyond that, the real question is ISP vs VPN.

ISP advantage over VPN:

- More regulated - Bigger, thus could have better focus on security - Less of a tasty target, because ISP customers do not specifically seek out to hide themselves, whereas VPN customers do.

Advantages of VPN over ISP

- Choice of jurisdiction (i.e. who can force the company to do stuff) - Company claims a focus on security - Choice of point where plaintext becomes available (for if you don't trust the beginning of your pipe)

I think this is a wash in general, but the jurisdiction point could matter if you don't like your local jurisdiction. Similarly, if you do like your local regulator, probably better to go with the ISP.

It's not like you're running a website on there - all you have to keep secure is SSH and your VPN. Keeping up to date with security updates and using a strong password (or better yet, private key) is essentially all you need to do.

I wouldn't necessarily recommend it to random non-technical people, but I figure most HN users could figure it out.

I've seen "infosec professionals" say they don't trust themselves to secure their own VPN server. It blows my mind...if you can't do that, why are you even in this business? How do you even use a personal computer?
I've heard the same thing from cryptography authorities in regard to rolling your own encryption and it makes sense to me. Having specialized knowledge opens your eyes to all the gotchas and gottahaves that most people wouldn't think about. While you certainly can take the time to set everything up exactly the way it should be and keep it updated, I'd rather spend my time doing/thinking about other stuff and am happy to pay the $120/year or whatever to let someone (vetted) else deal with it.
Rolling your own encryption is an entirely different animal. I agree with that sentiment for encryption.

When you use a VPN service, though, you really don’t have any real insight into what’s going on on their servers. Run your own and you can be sure you’re running the most recent, audited version of OpenVPN on an updated operating system.

You're right and I don't disagree with your core thinking, except to say that everyone draws a line beyond which they'll be happy to trade some risk for some time. I understand why someone would run their own VPN, but I also understand why someone wouldn't.
Yeah, I totally agree -- and there are a lot of very valid reasons someone might choose a service over a DIY solution.

My main point up there was just that I'm always kind of surprised and honestly have trouble believing it when I hear someone whose career is in network security say they don't trust themselves to secure a VPN server. If a person can tout creds securing an entire organization, a little ephemeral Linux instance that you can blow away and rebuild at will should be cake.

But this buys you nothing in anonymity to the sites you visit. The IP you use to access the internet is still a single IP tied to you via a billing account with your cloud provider. A commercial VPN service NATs your connection out with thousands of other customers.

So it all depends on what you want to use the VPN for.

I would look at one of the cryptoanarchist aligned providers like Cryptostorm, Mullvad, or AirVPN. Of course, no one is immune to a hack but they don't have any shady connections or financial incentive to deprioritize security.
That's good advice, but I'm not sure I fully understand - did NordVPN have such a financial incentive?
The best thing NordVPN can do right now is make a statement that clearly and honestly describes how its users are affected. No bullshit marketing language, no trying to hide facts, just a short and simple explanation of what this means for users and what they should do next.
Linked article says:

> “The server itself did not contain any user activity logs; none of our applications send user-created credentials for authentication, so usernames and passwords couldn’t have been intercepted either,” said the spokesperson. “On the same note, the only possible way to abuse the website traffic was by performing a personalized and complicated man-in-the-middle attack to intercept a single connection that tried to access NordVPN.”

> According to the spokesperson, the expired private key could not have been used to decrypt the VPN traffic on any other server.

> NordVPN said it found out about the breach a “few months ago,” but the spokesperson said the breach was not disclosed until today because the company wanted to be “100% sure that each component within our infrastructure is secure.”

If I were a NordVPN customer, these quotes would not give me confidence that my traffic wasn’t exposed.
> According to the spokesperson, the expired private key could not have been used to decrypt the VPN traffic on any other server.

It's simply not true.

Their CA private key which is used to issue certificates for ALL servers also leaked along with RADIUS key which is used to secure EAP session [1].

They DO hide facts.

[1] https://gist.githubusercontent.com/Snawoot/85f77356e229d77aa...

Truth is - if hackers did a MIM attack and collected a bunch user traffic (for how long?) they could have everything.. banking info, emails, logins...

at this point if i was a user of that VPN service - i'd be replacing all of my sensitive passwords, secret questions/answers to key accounts.

While I agree with you - I do think it's slightly less bad than you make it out to be. For example, if I connected to my bank over this VPN I would only be as concerned as my HTTPS connection. So my VPN still doesn't know my bank login, assuming my TLS was sound, right?

This would hypothetically be as bad as logging into my bank on a public wifi.

Am I paranoid enough to not log into my bank on a public wifi? Yes. So I should be concerned here. But, it's at least not immediately insecure.

TLS is probably not sound, judging by history, even if exploits aren't known. But apart from that, there may have been ways to infiltrate a target client from the VPN host and make their TLS moot.
MitM-ing a VPN does not break HTTPS. Hence, any passwords send over HTTPS are still safe. You could speculate that a VPN MitM is a nice way to get an MitM position for a further attack on TLS. But that requires a lot more speculation.

What isn't safe is your browsing history. True, any HTTP data isn't safe, but trusting that to be safe is baaaaad anyway.

In short. This leaked browser behavior, and could be a single step in getting a MitM possition on users.

NordVPN's advertising has deliberately downplayed the significance of HTTPS, as part of their fear mongering campaign about public wifi and residential ISP connections, so it's not really surprising to see such misconceptions raise their heads when NordVPN screws the pooch like this.
(comment deleted)
Only to the existent that any other ISP already could.

Banking info, email, etc would all be protected by encryption in transit (HTTPS or TLS), so an MItM attack shouldn't affect them. The attacker would only know what hosts you were communicating with, any unencrypted headers (ex. TLS SNI), but not the actual data itself.

A MITM attack of a VPN allows attackers to collect unencrypted traffic.

Most people access email over a webmail interface, like gmail, that uses modern TLS encryption. All that's sent unencrypted is the SNI header, e.g. "mail.google.com", and roughly how much traffic total is transferred, e.g. "20 MB of browsing on mail.google.com".

A VPN can't easily defeat TLS. It would require the user to ignore many scary warnings from the browser.

You're still right that a user should change their passwords for any websites that do not use TLS (very few these days), or for any that use old versions of TLS if their threat model includes someone with close to nation-state resources attacking their connections individually.

It also probably doesn't hurt to be paranoid and rotate anyway, but it should be with a proper understanding of the threats, not because of some ridiculous "the sky is falling" incorrect information like this.

The sky is not falling!

Everyone is discounting one thing - State actor possibility behind this attack.

With state actor comes completely different ball game - totally different budget and capabilities to crack things. NOBODY knows what their unpublicized capabilities could be! So it is good practice to stay vigilant!

Regular people who are NordVPN’s customers can’t possibly understand that. A highly technical explanation is not good enough. They need to put out a statement that explains clearly and concisely what this means for their users, something that all people can understand.
"To recap, in early 2018, one isolated datacenter in Finland was accessed without authorization. That was done by exploiting a vulnerability of one of our server providers that hadn’t been disclosed to us. No user credentials have been intercepted. No other server on our network has been affected. The affected server does not exist anymore and the contract with the server provider has been terminated."

Not sure what you're asking for? If someone doesn't understand that, then they probably aren't using a VPN.

Ok, I’m going to pretend that I’m a NordVPN customer who is 50, works as a plumber, and has installed a VPN on their phone because they were convinced that it’s very good for privacy. Here goes…

“What is a datacenter? How was it accessed? Like in that Mission Impossible movie? What are server providers? What role do they play in all this? Credentials? That’s like my passwords? What about my browsing activity? All I want to know is if my traffic was spied on.”

How was I?

Have you used Nord VPN? These aren't questions users would have. I could be wrong, but I'm confident that a plumber that has used Nord would know what a datacenter is, just from using the app. My tech illiterate Dad (70s) sure does.
VPNs are advertised a lot on all kinds of YouTube channels. It is reasonable to believe that they have a bunch of customers who know almost nothing about VPNs or the Internet in general.
Those are also the customers that won't even know that this hack ever happened, or won't care, because "everyone gets hacked anyways".

You said you wanted them to give a clear explanation, which I believe they did, but you don't. Let's just leave it at that.

These companies are giving all their customers the idea that they have reasonably good anonymity with a public VPN service, which is arguably false depending on how they use it. I feel like that's where you should be targeting this concern. Not an expired key leak.

This is always topical: Don't use VPN Services

https://gist.github.com/joepie91/5a9909939e6ce7d09e29

The article is slightly more nuanced... know when and why to use VPN is more accurate. As mentioned near the end of that article, using known or suspected hostile networks, like public WiFi is a good reason to use VPN.
The problems addressed by avoiding a locally hostile network by connecting to another, globally hostile network is solving a very limited, nuanced set of problems.

Unless you're VPNing to your home or office, these public providers are just asking for trouble. They're too cheap to run well.

Yes, you can run your own VPN... that’s a great solution.

Also public WiFi attracts low hanging fruit sort of exploits. Incentives for the VPN company that already makes money, to actively hack and exploit your machine are significantly less.

It’s not a privacy issue, it’s a security issue.

"Dont use VPN services"

then at the bottom: So then, what?

THIS TYPE OF VPN

Which is not a service, but a self-hosted open source solution.
Yeah much better, a server that the user will probably not be talented enough to secure and will forget to patch the OS, libraries or application itself. /s
Analogy: "Don't eat at restaurants if you want to control your food."

"If you for some reason cannot do that, here is a way to set up a food truck"

Analogies are imperfect, but I think the intention was more like "If for some reason you cannot do that, here is a simple meal you can make in your own kitchen."
> Because a VPN in this sense is just a glorified proxy. The VPN provider can see all your traffic, and do with it what they want - including logging.

So can my ISP and they have been confirmed to sell customer data and work directly with NSA.

https://en.wikipedia.org/wiki/Room_641A

https://www.theguardian.com/business/2016/oct/25/att-secretl...

Pick a cloud provider you trust. I was thinking of moving from Digital Ocean (US) to Hetzner (German) and setting my own VPN up through a normal server.
Why are public cloud providers more trustworthy than VPN providers? Some VPN providers are sketchy but not all of them.
It's not that they're more trustworthy it's that you have more control. They could be feeding traffic to the NSA as well but you can encrypt it yourself -- with VPN services like Nord you're relying on other people to do that for you but often VPN services can offer convenience services like country switching etc but if security is what you're after then cloud providers and setting up your own VPN seems like a more reliable alternative.

To the people looking to setup a simple http proxy in three steps:

1. Set up a server instance who's IP you know and have configured ssh.

2. In Browser: Manual SOCKS Proxy: 127.0.0.1: your_chosen_port

3. In terminal: ssh -i ssh_key -D your_chosen_port user@ip_address

I've pondered this before, but I don't see much advantage in having my traffic which currently comes from many IP addresses as I roam about the world, many of them shared and constantly changing, all come from one IP address that is absolutely only me.

Plus browsing the web from a hosting provider is a worse web; you'll get more sites rejecting you or putting you through bad CAPTCHAs all the time because the same service you can rent a server from, so can all the spammers and scrapers and other bad actors, so you're pretty likely to end up in an IP space with bad reputation.

If anyone can argue me out of this position, go nuts. I want this to work and do something useful, I just can't convince myself it does even with that bias.

This worse web is literally Google bullying you unless you tell them everything about who you are.
No, that's a different web. I live in the "google bullying" web between my combination of using Firefox + uMatrix on desktop, Brave on Android, and DuckDuckGo as my search engine. Google gets very little of my desktop info and fragmentary mobile use only. I do a few extra CAPTCHAs but it's not too bad.

The "I think you're a bad actor" web is much worse. Ask Tor users.

Sorry, I confused the two. I'm out here using Tor for my privacy (good kid; didn't do nuffin').
If you're in the US, how exactly does moving from a US host to a German host make you more secure?

At least there are a few shreds of controls remaining on US agency surveillance of US persons using US networks.

But there are absolutely zero controls on monitoring networks beyond US borders, so it's open season for non-US hosts.

>If you're in the US,

not just US location or even US services .. it's hard to be secure when we know that the US gov is reading and storing everythign they can.

In comparison -- the EU is not. The EU has the opposite approach and takes data privacy very seriously. This is backed up with effective legislation.

> Your IP address is a largely irrelevant metric in modern tracking systems.

I don't believe this for one second.

Your IP address on its own is not sufficient to identify you. That doesn't mean your IP address is not helpful in identifying you.

If you have Javascript disabled, it is a heck of a lot easier to identify you with a combination of an IP address, user agent, and OS than it is to identify you without the IP address cutting down the pool of potential visitors.

On top of that, if you're targeting me and do a geo-location of my IP address, it will get you within 5 miles of my house. That's close enough that you'll know which county I'm in, which with a few other easily-obtained pieces of information will let you pull up my voter registration, which will give you my exact street address.

Of course, you could mitigate this by setting up your own VPN on something like Linode, but unless you're regularly rotating IP addresses, you've just traded a pseudo-identifier that multiple people/devices share for a persistent identifier.

This argument comes up all the time, and I have never heard anyone explain it in a way that passes my sniff test. If you want me to stop using a VPN, you need to do a lot better than just claiming that IP addresses don't matter -- you need to show some kind of evidence to back that up.

If you have Javascript disabled, it is a heck of a lot easier to identify you because you're one of the very few who disabled Javascript.
Eh. If you're enabling JS because you think it's going to help you blend into the crowd, I am skeptical that you understand how powerful JS fingerprinting actually is, particularly around cache abuse and super-cookies.

You don't need to go all the way, but the very least I would advise turning on the resist-fingerprinting config in Firefox. At a minimum, block things like canvas/webGL. You're making yourself more identifiable by doing so, but the alternative is worse.

Now, if you're not using a VPN, and you're in a rural area, and you're on Linux/Firefox with Javascript disabled -- sure, I definitely buy that I could do some pretty decent correlation with that info. That's why VPNs (for all their flaws) still matter.

Sure, I do understand that, and yes IP hiding does matter. I'm merely pointing out that disabling Javascript (and eventually enabling some set unique to you, to un-break a broken site) is just another way to leak some bits one might want to be aware of. Faking the common fingerprinting vectors known to expose you uniquely is possibly a better way... until the new ones are found. I don't know. The leaking bits need to be carefully accounted for, and you don't know the site userbase for sure to blend into the largest cluster possible. I don't think that fingerprinting is something that can be fought by the end user efficiently, besides the very obvious things like blocking the major vectors.
That's a good point.

> Faking the common fingerprinting vectors known to expose you uniquely is possibly a better way...

I wish there was more research being done around this. I appreciate what Firefox is doing, and I assume there are good reasons for their fingerprinting strategies. They know more than me about this stuff. But... it still sets off some alarm bells in my head. It seems like it would be strictly better to spoof location/canvas/microphone data instead of only blocking it.

Yes and no.

Disabling javascript is like wearing a ski mask in a crowded mall.

It makes you much more obvious and easier to track, but harder to identify on the outset.

But who wants to put in the effort to develop tracking for non-JS users? In reality, most will just ignore the few users that don't want to be tracked. Even ublock origin should be enough for most.

However, IP is certainly used. I know of a few cases where IP is at least used as a filter. Most websites won't see that many users from one IP address.

> Your IP address on its own is not sufficient to identify you.

Wasnt there a story yesterday that FBI tracked some a guy who had logged into Jihadi forums with the IP, knocked on the door with a copy of a passport of the guy's dad.

It absolutely ignores government's censorship, though. I use NordVPN simply because I want to access resources that would be otherwise banned for me.
It also ignores that some VPN services have proven their no logging claims in court, multiple times.
More importantly, don't do anything private on a networked computer. As the daily breaches show, there is no such thing as computer security in 2019.
> You are on a known-hostile network

Which is precisely the use case I use a VPN for.

I'd rather trust an at least somewhat trustworthy VPN provider with my data than a random coffee shop and clients who happen to be on the same network at the time.

I feel like that's crazy. There should be no traffic entering or leaving your machine that's not end-to-end encrypted already. Trusting some fly-by-night VPN provider because they buy a lot of YouTube ads is no substitute for proper end-to-end session level encryption.
Simply seeing what servers you connect to can reveal a lot about you. Where you work, what social media accounts you have, what apps you have installed, what you are interested in reading etc. HTTPS doesnt help you with that.
This article tries to enumerate the use cases for use of commercial VPN services, but misses out my only use case of these services: evading geoblocks. It seems fallacious to me.
Exactly. Run your own. I setup and teardown vpn's all the time, takes about 5 minutes to launch, a min to teardown via simple cloudformation script.
A strongly-worded gist, but my sole use case is avoiding DMCA notices. VPN still seems like a good solution for that.
> The attacker gained access to the server — which had been active for about a month — by exploiting an insecure remote management system left by the datacenter provider, which NordVPN said it was unaware that such a system existed.

This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?

Sounds like an iDRAC exploit (assuming Dell servers).

But, yes, remote management is pretty common in datacenters. The fact that NordVPN wasn't aware of them just shows incompetence.

Depends on what kind. In case of idrac, yes; but it's weird that it was insecure by default in the first place. Usually credentials are configured and provided to the customer. Makes me think there might have been some other interface. Clarification is definitely needed.
How the hell do you pwn a server with iDRAC?
In certain configurations, iDRAC gives you an rdp connection. If idrac is left at default, windows admin login not being changed isn't too much of a stretch.
If you can reboot it without anyone noticing? Really, really easily: iDRAC gives you access to the local console, like a remote KVM. Reboot into single user mode, change a password, done.
Oh, IPMI and friends are a total mess. Some implementations allow one to take control of a running server remotely especially if they use a shared ethernet for management ( popular in supermicros ). I once had our security geek demonstrate it by taking over the running server, rebooting it using network emulated USB stick, adding a file into /etc and rebooting the server again.

In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.

The first time I booted a server using a virtual CD-ROM (iso on my laptop shows up as a hardware CD-ROM on the server) over IPMI I was simultaneously relieved (because I could fix the machine remotely) and absolutely totally horrified.
iDRAC is a full onboard whitehat rootkit manufactured and supported by Dell. It runs independently of any OS and has control over the system. It is intended to be a substitute for physical access.
User root, password calvin. That's the default. And, if I had a dime for every time I've seen one of these in a data center, I'd be a rich man. I have literally begged sys admins to change the default password, but they say, "Why... we're behind a firewall using RFC 1918 addresses. No one can get to these." The rest, as they say, is history.
> we're behind a firewall

This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.

In well maintained networks the management interface (IDRAC, etc.) for each server is placed on a separate VLAN which the servers cannot access. This isn't to say that cheap providers actually do this, or that the VLAN can't be accessed by a compromised technician's workstation/laptop.
So it's a fail-open design, given the rarity of well maintained networks, and the lability and inobservability of said state.

Never trust the network.

Yes, this kind of firewall is always supposed to be between the management hosts and everything else. Only the sysadmins at the data center a very limited set of applications is supposed to be able to access it. The very real risk is misconfiguration.
HPE iLO also had critical vulnerability: CVE-2017-12542.
I’d guess that the DC got owned, no need for iDRAC exploits when lazy VPN company staff never changed the pws.
Pointing fingers without having the details at hand is not competent either.
They mean IPMI. All servers have IPMI and there’s remote root exploits against many versions of them.
> All servers have IPMI

Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions.

It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).

I'm confused why a factually accurate post was down voted. anyone care to enlighten me?
Don't worry about it. It could have been a mistake or a bot randomly voting. Either way complaining about down votes is against community guidelines.
If they're going with a bottom-dollar host, it's possible that the out-of-band server management tools were exposed. It's less likely to be a software backdoor, and more likely to be Supermicro IPMI or other baseboard management controller.
Yes. Dedicated servers generally have IPMI / ILO / IDRAC / whatever. It's the only way to scale out management of hosts and provisioning.
To the person / people down voting, can you share your ideas for how you can go about providing at-scale management of large estates of physical machines? I've never seen anything else that's both as practical and as affordable as IPMI / IDRAC / ILO systems that ship with servers, that doesn't introduce weird new failure conditions that can impact significantly more than a single host.
Of course you have to worry about all of that! When you don't self host you have to assume whoever you are renting from hirers the lowest paid employees they can get to manage infrastructure for you. That's how they get profitable. You are not outsourcing expertise.
I know that public cloud providers like Rackspace and Azure insert their own accounts and services into cloud servers and VMs mostly under the guise of being able to support said servers and monitor them and their health.

True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.

Very few people go to "true data centers". Those are very expensive because you are buying power, space, cooling and cross connects. Racking machines, replacing hard drives, building a PXE-boot infrastructure, building a remote access infrastructure that bypasses the customer facing network is expensive and time consuming.
It doesn't make much sense to me, even with iDRAC/some other console access you don't really have access to OS unless you reboot & go to single user mode etc at which point they should be noticing their servers rebooting etc. would love more info
It opens an exploit chain, in a normal circumstance you are correct. In a malicious circumstance, it is always feasible irrespective of the likelihood.
Just set up your code as a boot-once config and wait for the owner to reboot their machine. Make your code end by booting the installed OS (or even by just rebooting again, most people will just curse about the damn slow server boot process).
You can't do that as you don't have any access until it's being rebooted. It's basically like you're standing in front of the machine so there's not really much you can do when you're just looking at a login prompt, you have to be able to stop grub from just booting with the default options and instead boot up using init=/bin/bash or maybe if the server supports iPXE you can just chain load some payload off the internet.
You can manipulate boot settings using BMC commands. No need to mess with Grub or the running system. Instead, tell the system to boot up from an emulated USB drive (image can be attached from some remote server, often including your web browser).

Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines).

Even SecureBoot won't help as you can just turn it off using the BMC.

See here for a bunch of examples for Dell machines using the BMC's HTTP API:

https://github.com/dell/iDRAC-Redfish-Scripting/tree/master/...

Why would they notice their server rebooting? And why would they not just assume it was a glitch or power failure?
When someone gets notified by their monitoring system that a server was unavailable (because it rebooted) they might investigate and see that the IPMI logs don't mention power loss

Power failure would require both of the power feeds in the DC failing simultaneously and would be easily verified by contacting the DC and asking if they had any power outages reported at the time. Of course there are cheapskates who don't go for redundant power supplies so it's possible but would be indicated in the IPMI logs

Servers can reboot for any reason. There are tons of kernel issues, especially since Meltdown & Spectre, that cause machine reboots in Linux especially on high traffic machines.

I've worked in production environments with thousands of machines and random reboots are a completely normal event for some workloads. Combination of hardware issues & kernel issues with hundreds of thousands of lines of code makes it inevitable. I would be surprised if NordVPN even noticed and their architecture wasn't designed to automatically start everything at boot.

You can't be perfect at scale - you just need to design your work loads to be redundant and fault tolerant.

Not sure if it is still the case but a while ago it was standard on OVH servers for them to put some public keys in their root-like's user authorized_keys. I think they used that to perform tasks requested from the web management system.
>> Is this common practice for data-center providers?

Absolutely. We had similar situation with one of the DC vendors.

So what can you do about it?
Use AWS/GCP/Azure
doesn't that shift the vulnerability/risk to the configuration of the firewall, security policy, and machine configuration?
Those cloud computing vendors know better than give open access to the word to your infra by default.
Full Disk Encryption is an option, but that’ll entail needing the use the IPMI console to enter the password at every reboot, essentially turning it into a manual operation.
If you have physical access twice, full disk encryption usually doesn't help really at all. IPMI seems about as powerful as physical access.
If you just ignore random reboots, sure. But why would you?
I don't think that'd be necessary, couldn't you make changes and then wait for natural reboots?
You couldn't, not via console access anyway.

However, if the attacker gains RCE many IPMI implementations theoretically allow for DMA, but this is a significantly more complex attack to mount in practice with no public PoCs available.

One can do LUKS with access via dropbear in initramfs.

If you're lucky during network install, you'll never need out of bound access.

But rebooting is still a manual process.

- Bring your own network and limit LOM/IPMI to a private network accessible by VPN only.

- Or, request a private network with no connectivity and a VPN device connected on that private network.

- Or, hardware disable LOM.

There's no reason you shouldn't request an ASA from your datacenter provider in this day and age.

I work for a web hosting company in the US and at least in our case, it's quite common for remote management to be enabled on pretty much all of our dedicated hardware. However, because of the inherent dangers in opening this up to the public internet, unless explicitly requested by the customer (or Managed Colocation), the NIC used for Dell iDRAC or HP iLO is on an isolated network unique to the physical datacenter. Remote access for our techs is managed through a secured bridge that requires all sorts of security hoops on our company intranet, and remote access for general internet traffic is not available due to the firewall restrictions. While it's plausible for remote access to be gained this way, it is extremely unlikely and would require several exploits at different points along the path.

I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.

IPMI does not have to be open to the internet to be open to a wide audience. Many of these out of band management interfaces are hosted on an internal network, but not isolated by customer.

Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies.

Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.

Which cheap data centers are you referring to? Curious as someone unfamiliar w/ the space.
Sorry hacker, not today!
Haha! If I was up to no good I wouldn't be using my real name in my handle to ask such questions :)
Generally speaking, there are four (4) tiers of "public" data centers are on the market, ranging from essentially a big room with some alright AC and a line out, to huge, highly secure (cameras, fingerprint readers, SSAE certifications, etc.) buildings with redundant power and HVAC systems.

The higher end ones are usually newer-ish, but there are lot of older "computer rooms" that offer acceptable-level benefits for a reasonable price. Lots of legacy customers and ISPs in these rooms, for the record. You get what you pay for but a lot of older DC spaces are just fine for most users; everyone thinks they need 99.999 but most don't.

There are also re-sellers and managed services companies who take out a footprint in data centers and then lease space in their cabs, sell bandwidth, IPs, etc. Using a series of resellers you can often get around restrictions as to what you're doing -- small fry MSPs don't ask a lot of questions -- but still get a data center footprint. These are sometimes one-man shows, and their quality and professionalism are often sub-par, which is how you end up with default iDRAC creds and the like.

Check out Data Center Maps or WebHostingTalk for some examples.

Source: data center ops manager for several different companies.

My social network for dog walkers definitely needs 5 9's. Thank you for the insight.
I have decided to contact them directly rather than publicize.
Maybe it wasn't open to the public Internet, but the VPN exit is inside the datacenter and connects out to the public Internet. Is it feasible that NordVPN provided their customers with a secure tunnel into their own datacenter's management software?
Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.

Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input).

Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.

> Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.

Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s.

And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.

.. AWS surprisingly doesn't.

Why is this surprising? AWS seem to know what they're doing in general, and this is obviously the right policy in this particular area.

How is it “obviously” the right policy? If implemented, console input would presumably be part of the AWS API, guarded with IAM permissions like everything else. If you have full IAM permissions, you can already take over any instance by temporarily attaching its disk to a different instance and modifying the data from there. (That requires rebooting, but so would takeover via console input.) Indeed, I’ve had to do exactly that on multiple occasions to fix broken config files on my personal instance; it would have been much more convenient to have console input.
This "personal instance" model doesn't really fit AWS: if the instance is borked then fail over to another one that isn't. No need for console input.
FreeBSD has an interactive kernel debugger that you can use with a serial console. Super useful to track down some things, not usable on AWS -- you'd need to (somehow) do a core dump and hope you can figure it out from there.
Amazon has security critical functionality on an unauthenticated http endpoint on a link local address. That's pretty damn dumb in my book.
That's a great link, but they're looking at "misconfigurations". How is that relevant here?
> Since the metadata service doesn’t require any particular parameters, fetching the URL http://169.254.169.254/latest/meta-data/iam/security-credent... will return the AccessKeyID, SecretAccessKey, and Token you need to authenticate into the account.

They talk about the AWS “Metadata Service” Attack Surface and how juicy a target it is. I was just providing support for that opinion.

Yes, software that runs on the instance can learn instance metadata. No, that is not a problem. Running e.g. user-supplied scripts on the instance would be "pretty damn dumb", but no one is that dumb. Any widely distributed software that did something shady with instance metadata would get busted PDQ. Just like any widely distributed software that did something shady with e.g. root credentials, which is about the same threat scenario.
It's crappy design which bypasses important security mechanisms of the OS (lower privileged users) by allowing every application with network access to access such critical functionality. One sane approach would be passing this information to the OS through the hypervisor which then exposes it as a properly ACLed file system.

This is like an author of a website vulnerable to CSRF (because it relies on IP for auth) blaming browsers for allowing cross site requests instead of require proper authentication. Except that Amazon is powerful enough to get away with pushing all the effort onto developers and admins.

Yes, in some situations it will take effort to not allow network access to untrusted software and/or users. For those situations, EC2 is not a good fit.
You can use iptables to limit metadata access to certain users but that takes effort so no-one does it.

I guess a machine-local service that takes ownership of the metadata service and implements additional restrictions (such as limiting access keys to privileged users) might be doable.

(comment deleted)
Leaving default creds on IPMI/iLO is not uncommon,some providers don't allow internet access,you have to login on the web console and use a java applet to access it iirc. I can't imagine a well reputed provider exposing ipmi to the internet but the nature of their business means they have to diversify server and network providers.
Typically data centers have compliance requirements like SSAE 16 specifies controls around physical access. Most any major retail data center would have that certification and others.

One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.

(comment deleted)
It seems NordVPN fucked up themselves are are now trying to avoid responsibility: https://www.theregister.co.uk/2019/10/21/nordvpn_security_is...

""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell.

"We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""

On Supermicro hardware and maybe others, IPMI has a very dangerous default setting: if you're not connecting the dedicated IPMI port to a network (typically some closed network dedicated to management), it will use the first ethernet NIC on the motherboard (sharing it with the host), possibly making it accessible through the internet (with default, insecure credentials adding insult to injury) or at least neighbouring machines.
Should be ok, after all they dont keep any logs right ?
NordVPN doesn't (or so they claim), but the hacker may have been logging activity.
If you're reading this and wondering which VPN service you should use to stay safe, start reading here: https://faq.dhol.es/@Soatok/cryptography/which-vpn-service-w...

(Spoiler: You're asking yourself the wrong question.)

You shouldn't have buried the lede.You should have just said, "Here is a link on why commercial VPNs are a honeypot and bad privacy."
Why does the linked suggestion say "Don't use an Android phone, use an iPhone instead."
This is so well timed, I just bought a 3-year subscription to NordVPN and they have a 30 day refund policy.
So did I, 33 days ago :/
You should probably ask for a refund, then set up your own VPN.

Commerical VPNs are, for the vast majority of cases, simply not a good bet for your privacy. You're changing your network traffic path from a diffuse and byzantine series of paths to once centralized collection point. The payoff for an attack on a VPN rises very quickly. Meanwhile, you're also conditioning yourself to say, "My traffic is secure while my VPN is on."

It's not a great combo.

I thinking about spinning up a Digital Ocean droplet and rolling my own right now
Absolutely. Look for project Streisand.
I'd keep in mind that cloud providers have well-known IP blocks that can sometimes be rate-limited by various internet sites/services, primarily to combat botting. You might inadvertently get caught in the IP range that's being actively rate limited by e.g. Instagram. YMMV.
It works for when you need to use untrusted WiFi, because the alternative is worse. Beyond that, it forms a nice defense against unsophisticated attackers. (e.g. it breaks a single datapoint (ip address) used by Google and FaceBook).
Is the alternative actually worse than SSL? Why?

And no, it doesn't break analytical by Facebook or Google in any substantial way. I know some people use them to evade Netflix region exceptions, and that's about all they're good for.

You can’t always ensure that all traffic goes over SSL. DNS traffic is an example. I always assume that hostile public networks like free WiFi have agents actively trying to man in the middle any connections they can. If your device has a known exploit and a single connection not going over SSL you drastically increase your exposure on a public WiFi, hence the one use case for VPN.
If your privacy concerns include your DNS requests then a commerical VPN isn't a realistic choice. And unlike some rando pseudo-bespoke brand-less coffee shop wifi, commerical VPNs are a big target.

> I always assume that hostile public networks like free WiFi have agents actively trying to man in the middle any connections they can.

And VPNs just move that problem. If you're not demanding and forcing SSL, you're not actually addressing this problem.

> If your device has a known exploit and a single connection not going over SSL you drastically increase your exposure on a public WiFi, hence the one use case for VPN.

I regret to inform you that none of these things you've described stop thise sort those attacks. Forcing SSL on your browser is a realistic option for most threat models. If you're at a level where you're actually being surveilled by a nation-state-level actor, a commercial VPN won't help you. Short of that scenario, forcing SSL will cover most cases.

Try running a traffic or packet monitor on a WiFi network. Now tell me how much of that traffic is going over SSL

And even if I don’t run my own VPN, I’d prefer to “move the problem”. It’s so much easier to attack machines on public WiFi than compromise a VPN provider... and much more anonymous, and less likely to incite law enforcement activity. Public airports, libraries, etc are hotbeds of nefarious activity.

I use plugins to force SSL to all connections. I block outbound non-SSL http traffic.

So, 0%? But personally I don't go to many sites that dont have full SSL coverage. Do you?

I highly recommend you do this.

> It’s so much easier to attack machines on public WiFi than compromise a VPN provider... and much more anonymous, and less likely to incite law enforcement activity.

Do you think there will be a successful law enforcement follow up to this breach? I doubt it.

> Public airports, libraries, etc are hotbeds of nefarious activity.

As are VPN data centers, as evidenced here.

If you really want to just shift your egress point, lots of self-hosted VPN options exist. These are much better able to do the things you want to do, without being as vulnerable to corporate VPN attacks.

Well I agree with you on the self hosted VPN option being the best. I’m just saying there’s not zero benefit to hosted VPN in some cases.

And it sounds like your just looking at http traffic and web browser traffic. Your computer is communicating over lots of other ports and protocols that are often not encrypted. Are you blocking all outbound traffic?

Let’s take the recent iTerm vulnerability. ( https://www.kb.cert.org/vuls/id/763073/ ) I’m guessing you don’t have a plug-in to force curl to use https? What if you execute a script that curls http and you don’t realize?

Now you could say well “I just make sure all my curls are https.” The problem with that approach is it requires unrealistic levels of vigilance, about every outgoing service you may use, and that all your software on your machine is patched or bug free.

The easiest and quickest place for a hacker to learn their tools and skills is simply public WiFi. Want to try that iTerm exploit out... you go to the coffee shop and wait for a programmer to accidentally curl something over http.

VPN is not perfect, but it does provide some protection in certain circumstances that can’t be ruled out.

Best course, force https for web browser, and use your own hosted VPN anytime you are on a public network.

>You can’t always ensure that all traffic goes over SSL. DNS traffic is an example.

But wouldn't you get a cert error if someone messed with the DNS to send you to a different IP than you would normally?

(Especially if they're using pinned certs, which many sites do now)

But how can a DIY VPN serve you if you want to, say, avoid geoblocked pages? usually with those VPN services you can choose where your exit node is. I don't feel like using a multi-region setup for this (well, now that i think of it using Terraform + Algo it could be neatly automated...)
Run a vpn instance in a different region? Lots of providers do this and it's fairly inexpensive for burst use.
Prepare to have to deal with their "Customer success team", I had to email a few times back and forth before they actually closed and removed my account. There is/was no way to do this in the web interface itself. This was over a year ago though so I don't know if they still operate that way.

I got a few good laughs out of that signature :)

>NordVPN said it found out about the breach a “few months ago,” but the spokesperson said the breach was not disclosed until today because the company wanted to be “100% sure that each component within our infrastructure is secure.”

So instead of allowing their customers to do their own damage limitation, they left their customers in the dark and continued to expose them to a breach they weren't sure they had fully contained.

I wonder when that sort of thing will become a criminal offence.

Yeah, that sounds like flagrant incompetence
Calling it incompetence lets them off the hook I think. This was a deliberate choice to keep customers in the dark, which is worse IMO.
Yeah, the word for this is 'negligence'. Sounds like someone should get sued at the very least.
Within 72 hours According to GDPR I thought? “ The GDPR introduces a duty on all organisations to report certain types of personal data breach to the relevant supervisory authority. You must do this within 72 hours of becoming aware of the breach, where feasible.” https://ico.org.uk/for-organisations/guide-to-data-protectio...
Maybe NordVPN would argue personal data wasn't breached?

It's a bad look in any case.

Theis Keys were stolen. All data is exposed now.
Only if someone used it successfully for MITM attacks. We don't know that, they can still argue that in fact no user data got breached.
I don't think they care about GDPR much. They were set up in a way to avoid legal scrutiny (not a bad idea for a VPN provider).
> I wonder when that sort of thing will become a criminal offence.

If they have EU customers then article 33 of GDPR should see to that.

"In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons. Where the notification to the supervisory authority is not made within 72 hours, it shall be accompanied by reasons for the delay."

Unless the authorities in EU accept the explanation they are in trouble, but I think you shall report even if you think there has been a breach.

They could argue that they don't have proof that the exposed private key led to a personal data breach.
Yeah, I think that's their only option actually.
Exposing a private key is basically a breach.
"breach" usually has a legal definition, which varies by jurisdiction
They would have to be sure that no private data leaked though, right?
> to the supervisory authority

Does this always/necessarily lead to customers/the public being informed? But yeah, better than nothing.

(comment deleted)
I think that is pretty criminal already.

Basically:

1- Nord falsely blames its server provider.

2- Nord hides it from their users.

3- Nord claims all will be well with an “audit” (again, since they were already “audited”)

This is either criminal negligence, “security theater”, or both.

> Nord falsely blames its server provider.

I don't see anything in the article about those claims being false. Where did you get that?

From the article: "“One of the data centers in Finland we are renting our servers from was accessed with no authorization,” said NordVPN spokesperson Laura Tyrell."

I believe that would be the section they're referencing.

How does this quote demonstrate that the service provider was not at fault?
“We failed by contracting an unreliable server provider...”

They are casting blame on the provider. Providing remote access tools is not a fault. Failure by NordVPN to disable said access is the issue, yet they passed the blame on.

Okay that makes sense. I read differently, like they had fabricated the story about the service provider's management console.
In Australia under the Consumer Data Right it would be an offence to wait before disclosing this. Unfortunately it doesn't cover all industries yet.
Sorry for posting under top comment, but I think it is very important.

Official response hides fact OpenVPN CA keys also leaked, so attacker could impersonate any other NordVPN server: https://gist.githubusercontent.com/Snawoot/85f77356e229d77aa...

RADIUS secret key also leaked, so propably it is possible to break into EAP session which infers session secret key for StrongSwan.

Also allowing historical sessions to be decrypted.
Also looks like NordVPN has been misleading customers about the number of servers they have (or didn't make clear they were VM/containers).
I don't think they ever wrote anywhere that they have 6,000+ physical servers. Calling a VM (like an EC2 instance) a server is not unusual. For the customers it was important that the resources, bandwidth and different IPs were available. For that it doesn't matter if it's a physical server.
Who has logs of full historical sessions, and the tools to decrypt them? NSA?
Yeah, as far as I understand that's pretty much the purpose of the Utah datacenter.
(comment deleted)
> RADIUS secret key also leaked, so propably it is possible to break into EAP session which infers session secret key for StrongSwan.

Could you elaborate on this? I am familiar with PKI so the first part makes sense, but I am not familiar with the intricacies of VPNs so I am not sure what this means.

When StrongSwan EAP-RADIUS plugin is in use, authentication delegated to RADIUS server. Actual EAP handshake occurs between VPN client and RADIUS server. [1]

Some EAP authentication methods (namely, EAP-MSCHAPv2 and EAP-TLS) export Master Session Key, which is exported to StrongSwan via MS-MPPE-Send-Key/MS-MPPE-Recv-Key EAP attributes. [2] So, MSK derived on RADIUS side and sent to StrongSwan. Eavesdropper with knowledge of RADIUS secret key capable to intercept and decrypt such EAP payload.

[1] - https://wiki.strongswan.org/projects/strongswan/wiki/EAPRadi...

[2] - https://tools.ietf.org/html/rfc5216#section-2.3

Read up on "Perfect forward secrecy": https://en.wikipedia.org/wiki/Forward_secrecy

Assuming their IPsec was enabled with it (and OpenVPN should be enabled by default), them leaking their keys does not matter. The sessions can not be decrypted even if the master key is leaked.

TLS also has perfect forward secrecy by default.

Impersonation is an issue, but the article stated the CA keys have already been rotated and are out of date.

EDIT: I meant to reply to the post below me, but this is fine. Sorry about that!

> Impersonation is an issue, but the article stated the CA keys have already been rotated and are out of date.

They were not rotated back then in 2018, so we can only guess if MITM had place. Their line of defence appealing to keys which are NOW outdated is just ridiculous.

> Read up on "Perfect forward secrecy": https://en.wikipedia.org/wiki/Forward_secrecy

> The sessions can not be decrypted even if the master key is leaked.

It's not true. PFS provides cryptographic isolation between long-term keys and session key used to encrypt data. Obviously, if MSK compromised it is irrevelant, how it was inferred: with PFS or not.

> They were not rotated back then in 2018, so we can only guess if MITM had place. Their line of defence appealing to keys which are NOW outdated is just ridiculous.

MITM could have taken place anyway because the attacker was on the machines. They did not need the key.

> It's not true. PFS provides cryptographic isolation between long-term keys and session key used to encrypt data. Obviously, if MSK compromised it is irrevelant, how it was inferred: with PFS or not.

PFS implementations have the session key rotated automatically in software and dependent on the implementation multiple session keys are in use at any given time dependent on flow. The PFS session key would also be different for every VPN server in the NordVPN environment. The only possible way to compromise a session on another VPN node (that was not compromised itself) would have been to intercept it at the time of the session being created and MITM by injecting your own PFS session key.

That is why it is called "Forward secrecy": the session can not be decrypted in the future, only in the present.

Unless your assumption is that this is a state actor with the ability to MITM connections in the first place, or a rogue ISP BGP hijacking that would have been obviously seen on something like BGPStream [https://bgpstream.com/], it is safe to say that no other VPN node's traffic was compromised. Only traffic on this single host in this single location.

More reading: https://www.speaknetworks.com/what-is-ipsec-vpn-pfs-perfect-... (Step 5)

"Instead of making use of the DH Keys Calculated during Phase-1, PFS forces DH-Key calculation during Phase-2 Setup as well as Phase-2 periodic Rekey. The PFS ensures that the same key will not be generated and used again."

OpenVPN (using TLS) also uses PFS by default. There is a reason it is called "Perfect."

EDIT: PFS is commonly also implemented by Diffie-Hellmen (DHE) key exchange: https://en.wikipedia.org/wiki/Diffie–Hellman_key_exchange

EDIT 2: I am not defending them as well. I just believe extrapolating the technical details is fear mongering at best. Believe it is best to focus on the facts as it makes for a stronger argument.

I've lost track of discussion and maybe some misunderstanding takes place, but I'll attempt to synchronize.

There are two distinct severe security issues. First one is leaked CA key, which allows to certify any key as a valid key for NordVPN server certificate key. I think it is not necessary to argue about this: traffic decryption to any Nord OpenVPN server became simple as network MITM. Not likely a state-level BGP hijack, but local attack targeting channel of small group of users. Anyway, we do not have cryptographical guarantees since this point.

Second issue is leaked RADIUS key. Why it is a problem for encryption? Because EAP authentication and key derivation runs between VPN client and RADIUS, and VPN server receives derived keys as attributes of EAP message: https://wiki.strongswan.org/projects/strongswan/wiki/EAPRadi...

> The eap-radius plugin does not implement an EAP method directly, but it redirects the EAP conversation with a client to a RADIUS backend server. On the gateway, the EAP packets get extracted from the IKE messages and encapsulated into the RADIUS protocol, and vice versa. The gateway itself does not need special support for a specific EAP method, as it handles the EAP conversation between the client and the RADIUS backend more or less transparently.

> For EAP methods providing an MSK, the RADIUS server must include the key within the MPPE-Send/Receive Keys; Unfortunately, FreeRADIUS before 2.1.10 did not include these attributes when used with EAP-MSCHAPv2.

So, despite session encryption key is not bound directly to long-term secrets which server possesses, they can be extracted from communication between StrongSwan and RADIUS server.

PFS has nothing to do with all of it. In first case it is possible to issue VALID certificate for eavesdrop VPN server and redirect users traffic to it. In second case MSK probably can be extracted communication between VPN server and RADIUS server (I can't say if it will require MITM of RADIUS session or it is possible to decrypt EAP payload with passive sniffing and posession of RADIUS secret key).

The tinfoil hat would argue maybe this was a leak that happened, but it was shared by design. It’s an HK company with questionable relationships and owners.
Maybe they only disclosed it publicly but notified customers out of the public eye ?
I was a NordVPN customer 4 months ago and got no notification. I’m hearing about this now via this post on HN. Dropped them around July.
i find it surprising that none of the threads in this topic mention the very serious threat this breach might have for users in countries like china. the fact that nordvpn neglected to tell its users for months after the breach quite possibly endangered people's lives. unforgivable.
The interesting thing about OOB on most modern servers is that its a separate, physical NIC. Not only is that easily VLAN able, a more security conscious datacenter could even air-gap the out of band LAN!
> The interesting thing about OOB on most modern servers is that its a separate, physical NIC. Not only is that easily VLAN able

On lower grade servers OOB is using main NIC. It's still possible (in all implementation I have seen, which is not too many) to have OOB in VLAN.

> a more security conscious datacenter could even air-gap the out of band LAN!

1. If you air-gap remote management, you take away it's function.

2. It's not possible to truly air-gap OOB if servers with OOB are not air-gapped (it's theoretically possible to use server to get into OOB network by exploiting/flashing custom OOB from OS).