I asked myself, why on earth do websites have to be slow when they're under heavy load?
To be fair, the technical reason is that the modern web (circa last 3-5 years or so?) has mostly switched over to demand scaling, and this is why hug-of-death situations are comparatively rare now.
If you have to be super-economical, and even setting an upper spending limit is uncomfortable, set your site up to ping you when/if it's falling apart so you can pull the trigger on ordering more cores yourself. Maybe put a super easy "order more cores" button in the ping, make 2:30AM disasters less terrible.
Of course, technical reasons aside, my initial unimpressedness (or, I must admit, indignation - ha) was directed at whoever this group's ISP is. Clearly the ISP is not providing enough [demand] bandwidth.
[Edit: After reading some comments below I think I should insert a preface/prefix here and say that it's entirely possible the website itself is misconfigured. I think I got sufficiently indignant that I didn't factor this in, and this was definitely an oversight. I don't know the situation, and it's unfair to squarely blame the ISP. Original message continues unedited...]
So, FWIW, this group is using mediatemple. (Found by a simple google of the IP and careful untangling of the results.) ...Well that's a name I've not heard in years. And it made me wonder - do MT have an answer to demand scaling? I'm not able to discern either way from a quick browse of their website (including the "managed wordpress hosting" section). If they don't (and they seem to be stuck in the VPS days), ouch. If they do have demand scaling, then I think it's unprofessional to offer configurations that don't handle situations like this.
In any case, a "504 Gateway timeout" with a tiny "nginx" underneath is going to make people go "hmm, who's this person's ISP" and knock a few points off whatever the answer to that question is. I wouldn't want that happening to me or anyone I build websites for.
[Edit, in the same vein as the edit above: the conclusion above is plausible, but incomplete. Again, it's necessary to consider all angles.]
I'm always wondering the same. As a pretty noob sysadmin, my dead cheap[0] second-hand dedicated server was able to handle being on the HN homepage for 11ish hours without breaking a sweat. No CDN was involved what so ever, all content was going directly to my single point of failure.
Yet, I have to resort to looking at cashed versions of the submissions from companies that have way more resources than I could ever imagine to have.
[0] Price of the cheapest Media Temple VPS offering.
Looking at this page's source it seems like Wordpress (or rather Shitpress) is involved, so this downtime is not surprising at all considering how many resources that pile of garbage uses.
But yeah, a properly optimized site can handle being on the HN front-page on a single server with no issues.
Would you buy a new car if it needed duct tape just so it can stay in one piece while driving on the highway? So why would you think it's acceptable for a CMS to require third-party "cache" plugins just to be able to handle traffic?
Computers and cars are both solutions to problems, but the complexity of computers is yet to be routinely contained without targeted effort. This is especially true of the Web. Wordpress is the single most installed CMS (and kitchen sink, at this point), it's a massive attack target, and it's installed by people who don't really know what they're doing.
Between being a kitchen sink (and having to deal with the internal/external API/architectural [compatibility] baggage that comes with any long-lived implementation) and having to be the CMS equivalent of Windows/macOS, it's a... difficult ask for a WP installation with (say) 30 poorly-constructed addons to to run smoothly. Possible, yes, absolutely, but it'll need tuning.
Caching is simply the simplest bandaid that can be tacked on. If the caching is tuned properly, it'll fix everything else.
Yes, this will mean the underlying configuration/state will be the digital equivalent of a giant wound you just want to clean out and fix up, but, err... it works.
The vast majority of WordPress-powered sites have no personalised content: the only authenticated users are 1-2 administrators, and 1+ people who add/edit content.
For this use-case, it's hard to beat WordPress for features and ease-of-use, especially when you consider the ecosystem of plugins that can add commonly-needed functionality. Sure, you could roll-your-own, and then add some code to generate a sitemap.xml etc., but there's no need.
Sure, it doesn't scale without adding caching, but it's trivial to add caching with either a plugin, or by putting varnish or similar in front on the same machine, or using a CDN.
Sure, if you have dynamic or personalised content, you'll need to cache further down the stack (e.g. at the database level), and that's less efficient (because you're probably running multiple php files to generate a single HTTP response), but that's not the usual situation.
That analogy really has no real relevance to the discussion as far as I can see. Caches help dynamic sites scale up to meet large amounts of traffic. If you don't want to use a WP plugin then use Varnish instead.
Hrm. Very well put :) thankyou for this perspective. Highly appreciated.
I hadn't really factored in website optimization. Who knows if each page load is chewing CPU for a couple of seconds because of an unoptimized query. I was only reading the other day about a random situation along those kinds of lines...
For 99% of sites, a simple CDN solves all scaling challenges. And it's cheap. Even a 1-second cache timeout can drastically reduce the biggest load spikes.
I don't think it should matter who simple falsifiable statements of facts come from. 'Adversarial forces' would be the ones most motivated to discover these bits of information, and they're incredibly valuable. I mean this is an experiment anybody with a couple of phones and a dash of technical ability could carry out themselves.
I would contrast this against the sort of macro-level research that involves modeling and other more subjective components subject to extensive 'tuning.' For instance the 'Uber drivers earn $3/hour' piece that ran a while back. The extensive modeling and obscured assumptions make it much easier to create fake conclusions that are very difficult to falsify.
It matters because the manner in which any "findings" are presented has a big role in shaping public opinion and in this case the opinion shapers are purpofuly distorting finding to attack an advesary.
See the WSJ's false report that google let's developers read user emails, it's technically true but that only happens after the user allows it via an explicit permission screen.
Or the AP's claim that google keeps track of user location ever after location history is turned off, which again is technically true but that is because location history is just a location feature and not the master switch for all location features.
In both cases other media outlets repeated the claims without further commentery of research and this cartel might be the reason.
Wasn't the entire point of this location privacy redesign Google did a few years ago that they wanted to have a single master switch, for internal and consumer clarity?
Because that's the line they sold us inside the company when we complained that it was now asking for broad permissions too often. "It's broad, but it's all controlled in one place now."
If they're now claiming that users are confused and that's not what it means, well, fuck Google management. They don't give a damn. They'll lie to their employees, and they'll lie to their users.
Even that seems to get periodically re-enabled on my Galaxy S7 without my permission. I reasonably expect that if I launch an app that requires location to get asked for it, but in my experience the system level software is collecting it anyway, perhaps traffic mapping?
Given Samsung, I wouldn't be surprised if that issue was related to their software. There's just so much going on in Android, and so much third party software, that you really can't assume much anymore.
I do and the only time you'll get a straight word from leadership is when Sergey appears on stage drunk (which is often. dude has a problem). The rest are consistently awful.
Ok, since you claim that "the opinion shapers are purpofuly distorting finding to attack an advesary", name one false claim from the PDF document's executive summary. Should be quite easy, right?
You're letting Google really easy of the hook regarding the location history PR disaster. Google uses dark UX patterns to trick their customers into sharing more data, and this is one more instance of that. The fact that "technically" location history is not a master switch is just a flimsy excuse.
Can I name some tautological claims instead? The last three bullet points are all basically conjecture. "Google could do this" or "Google has the capacity to do that".
The article uses dark patterns to make you think they're making strong claims when they're just saying things that might be the case, or that the researchers don't have the ability to rule out. And it doesn't help that this article goes to great lengths to word things in more nefarious ways than the underlying research paper.
There’s nothing that makes the location history issue only ‘technically true’. It is simply true.
If you present a switch saying don’t record location history, and still record the history, you are not respecting the switch.
Saying “That switch is just to turn off tracking the location history for the end user to use. We’re still going to track them for our own purposes.” is fine, but it’s a lie to say that the switch turns off location history. The switch should make it clear that all it’s doing is turning off the user facing version and that Google is going to keep tracking you whether you like it or not.
That would be honest, but presumably people wouldn’t like it if they knew, so Google wasn’t truthful about it.
The reporting is true, and Google deserves the consternation they are getting for misleading users.
Most if not all research that are widely pitched to the media have some sort of agenda driving them.
In this case we have invasive surveillance from Chrome and Android which is not a secret and cause of growing concern.
As is deceptive communication about 'improving user experience' which is meaningless without specifics and is intentionally used to mislead, and downright fraud with 'location buttons' that do not turn off location.
The fact that you didn't seek to engage these points and immediately sought to question motives betrays an agenda.
Unfortunately these are unethical actions whoever does the research and this kind apologism seeks to derail discussion and distract from the creepy behavior of bad actors.
Nice way of missing the point. I don't expect MS will ever stop. There's no incentive, people already proved that there's no treatment so bad that they will put their foot down and demand a change.
Google opened the way for collecting massive amounts of data by any means possible (including and not limited to tricking the user into thinking they aren't). And people seem pretty at ease with this, comparatively few are as angered by the revelations these days as they were about MS's relatively harmless telemetry.
So going back, at some point you'll miss the times when you thought MS collecting data about your hardware and drivers was "the evilest evil".
It might just be me, but I believe most people thought Google was a worse violator prior to Windows 10. Some (probably naive) people used to think Windows was the last bastion of non-Linux hope.
Not really. MS has (had) a very crappy image that they worked really hard to build during the early 2000s. That reputation stuck. And it doesn't help that most people are ignorant and are always willing to ignore an informed opinion over one that sounds cool.
MS and Windows 10 are still far behind Google in the data collection game. Especially since they put that privacy wizard front and center at the first setup.
But as I said, every second you spend defending Google and throwing shade at MS because it feels like the cool thing to do, Google adds another data collection item in their policy, hides yet another setting under 5 layers of options, and tricks you again into thinking that toggling a switch to off actually disables it.
It's not like they did exactly that for 10 years now to the point where they are by far the largest private data collection effort in the history of man kind.
And yes, I'm sure MS will follow suit. Why not? Google seems to be doing just fine.
You don't need to convince me! I think Microsoft is just as evil as Google and vice versa! Microsoft did their best to ruin personal computers and Google continues to try to ruin cell phones.
Luckily, in both cases, free/open source software exists to fix both PC's and (some) cellphones!
It would seem that you're the one missing the point. Microsoft was the very first company to sign up with the NSA way back in 2007. So when it comes to the mass collection of user data and spying - Microsoft is a pioneer in the field.
Take a look at the privacy settings in your Microsoft account, specifically the activity history. They collect very much the same stuff that Google does, only Google lets you disable the data collection and Microsoft doesn't.
I am reasonably sure this is not the case, especially since every piece of evidence, article, and comparative study points at Google as being by far the worst offender on the data collection and privacy topics. If you have an Android phone you'd better give up on the idea that you can really "disable" the data collection.
Windows on the PC gives me clear options immediately after installation to disable all that and what's left is the "basic" telemetry, which is very light data collection according to today's standards. At least there's nothing personal there. Also I can very well use the PC without a MS account (so far).
On the mobile side... Is MS still into that? Myself along with probably 5 other people do have a Windows 10 mobile phone but again, the privacy wizard apparently did a great job to disable most of the collection of data. And when I go to Settings > Privacy > Location and I disable it it's disabled.
But yeah, this is the kind of ignorant BS that got all of us here in the first place. As long as there's a critical mass of ignorance on the market the likes of Google know they'll always have someone to sell to so they shove this down everyone's throat. MS makes money by selling you the products. Google gives you everything for free. Not raising any flags?
Please go to https://account.microsoft.com/privacy/activity-history and see for yourself. You can delete your activity history (including your search history) but you cannot disable the data collection as you can in your Google account.
>MS makes money by selling you the products. Google gives you everything for free. Not raising any flags?
Microsoft makes money in all sorts of ways, among them free and ad funded services like Bing, Skype or LinkedIn.
But yes it does raise flags. That's why - as a paying Office 365 (for Mac) subscriber - I was expecting to have more control over data collection than as a user of ad funded services, not less.
I think you really want to miss the point: The problem isn't that Google is collecting performance data and other such stuff. They collect very private data that can be directly tied to you - fauigerzigerk personally. And they make it very hard or impossible for you to really disable it.
First, MS doesn't exist on the mobile side. This might sound irrelevant until you realize that a phone has the potential to be a much more powerful data pump.
Second, most Windows installations aren't tied to a MS account. As such no data can be linked to it and to you in person.
Third, you see this screen (a few settings are off-screen)? [1] It actually disables most data collection, like location. It doesn't just hide it from the dashboard and have you disable it "again" in 3 more places like Google was shown to do these days. Just by disabling Cortana you get rid of most data collection that can be tied to you.
This is the first thing I do on any Windows machine. Between this and the fact that I don't have to log in with a MS account means that my MS activity dashboard only includes my explicit browser logins and this:
We don’t have any data associated with this Microsoft account at the moment.
Or this:
There’s nothing to see here yet. To add some interests in this category, open Cortana’s Notebook on your device.
>I think you really want to miss the point: The problem isn't that Google is collecting performance data and other such stuff. They collect very private data that can be directly tied to you - fauigerzigerk personally
And here's the point you are in fact missing or rather denying: Microsoft does collect tons of very personal data as well, not just performance data. You are acknowledging as much when you say that disabling it is the first thing you do on any Windows machine.
Most people will never change the defaults, and Microsoft is pushing extremely hard to make all users of their software log in to a Microsoft account. A lot of Office 365 functionality makes no sense without being logged in. So it is important what privacy settings there are, what the defaults are and what you can or cannot disable.
But I think the reason why we are talking past each other is that I don't use Windows. I don't have the settings in your screenshot. I was comparing the settings in my online Microsoft account to the settings in my Google account.
If your main point is that Google is far more dependent on collecting as much personal data as they possibly can than Microsoft, then I fully agree with that. But that makes it all the more baffling why Microsoft behaves so much like Google when it comes to data collection.
In some ways not surprised, but its still a LOT of data (in sheer volume). Does Google even keep the data and analyze it, or is much of it thrown away as digital exhaust? Anyone know?
My favorite is trying to trace the problems back to Larry/Sergey, which is why I wrote the essay. Worth mentioning that one of the hardest parts of writing it was tracing the history of things like Google Analytics.
52 comments
[ 5.4 ms ] story [ 55.7 ms ] threadhttps://webcache.googleusercontent.com/search?q=cache:m-ItC3...
To be fair, the technical reason is that the modern web (circa last 3-5 years or so?) has mostly switched over to demand scaling, and this is why hug-of-death situations are comparatively rare now.
If you have to be super-economical, and even setting an upper spending limit is uncomfortable, set your site up to ping you when/if it's falling apart so you can pull the trigger on ordering more cores yourself. Maybe put a super easy "order more cores" button in the ping, make 2:30AM disasters less terrible.
Of course, technical reasons aside, my initial unimpressedness (or, I must admit, indignation - ha) was directed at whoever this group's ISP is. Clearly the ISP is not providing enough [demand] bandwidth.
[Edit: After reading some comments below I think I should insert a preface/prefix here and say that it's entirely possible the website itself is misconfigured. I think I got sufficiently indignant that I didn't factor this in, and this was definitely an oversight. I don't know the situation, and it's unfair to squarely blame the ISP. Original message continues unedited...]
So, FWIW, this group is using mediatemple. (Found by a simple google of the IP and careful untangling of the results.) ...Well that's a name I've not heard in years. And it made me wonder - do MT have an answer to demand scaling? I'm not able to discern either way from a quick browse of their website (including the "managed wordpress hosting" section). If they don't (and they seem to be stuck in the VPS days), ouch. If they do have demand scaling, then I think it's unprofessional to offer configurations that don't handle situations like this.
In any case, a "504 Gateway timeout" with a tiny "nginx" underneath is going to make people go "hmm, who's this person's ISP" and knock a few points off whatever the answer to that question is. I wouldn't want that happening to me or anyone I build websites for.
[Edit, in the same vein as the edit above: the conclusion above is plausible, but incomplete. Again, it's necessary to consider all angles.]
Yet, I have to resort to looking at cashed versions of the submissions from companies that have way more resources than I could ever imagine to have.
[0] Price of the cheapest Media Temple VPS offering.
But yeah, a properly optimized site can handle being on the HN front-page on a single server with no issues.
Between being a kitchen sink (and having to deal with the internal/external API/architectural [compatibility] baggage that comes with any long-lived implementation) and having to be the CMS equivalent of Windows/macOS, it's a... difficult ask for a WP installation with (say) 30 poorly-constructed addons to to run smoothly. Possible, yes, absolutely, but it'll need tuning.
Caching is simply the simplest bandaid that can be tacked on. If the caching is tuned properly, it'll fix everything else.
Yes, this will mean the underlying configuration/state will be the digital equivalent of a giant wound you just want to clean out and fix up, but, err... it works.
:/
For this use-case, it's hard to beat WordPress for features and ease-of-use, especially when you consider the ecosystem of plugins that can add commonly-needed functionality. Sure, you could roll-your-own, and then add some code to generate a sitemap.xml etc., but there's no need.
Sure, it doesn't scale without adding caching, but it's trivial to add caching with either a plugin, or by putting varnish or similar in front on the same machine, or using a CDN.
Sure, if you have dynamic or personalised content, you'll need to cache further down the stack (e.g. at the database level), and that's less efficient (because you're probably running multiple php files to generate a single HTTP response), but that's not the usual situation.
I hadn't really factored in website optimization. Who knows if each page load is chewing CPU for a couple of seconds because of an unoptimized query. I was only reading the other day about a random situation along those kinds of lines...
Nice article worth reading even though I got a few 504. It looks more professional when the webpage loads in a second on first try.
http://digitalcontentnext.org/membership/members
http://digitalcontentnext.org/about/overview/
At the very least it reflects poorly on their member's coverage of FB/google.
I would contrast this against the sort of macro-level research that involves modeling and other more subjective components subject to extensive 'tuning.' For instance the 'Uber drivers earn $3/hour' piece that ran a while back. The extensive modeling and obscured assumptions make it much easier to create fake conclusions that are very difficult to falsify.
Are you just trying to smear the information with an ad hominem, or do you have reason to believe it is inaccurate?
See the WSJ's false report that google let's developers read user emails, it's technically true but that only happens after the user allows it via an explicit permission screen.
Or the AP's claim that google keeps track of user location ever after location history is turned off, which again is technically true but that is because location history is just a location feature and not the master switch for all location features.
In both cases other media outlets repeated the claims without further commentery of research and this cartel might be the reason.
Because that's the line they sold us inside the company when we complained that it was now asking for broad permissions too often. "It's broad, but it's all controlled in one place now."
If they're now claiming that users are confused and that's not what it means, well, fuck Google management. They don't give a damn. They'll lie to their employees, and they'll lie to their users.
There's a master switch it's just not what the AP decided it was.
You're letting Google really easy of the hook regarding the location history PR disaster. Google uses dark UX patterns to trick their customers into sharing more data, and this is one more instance of that. The fact that "technically" location history is not a master switch is just a flimsy excuse.
The article uses dark patterns to make you think they're making strong claims when they're just saying things that might be the case, or that the researchers don't have the ability to rule out. And it doesn't help that this article goes to great lengths to word things in more nefarious ways than the underlying research paper.
If you present a switch saying don’t record location history, and still record the history, you are not respecting the switch.
Saying “That switch is just to turn off tracking the location history for the end user to use. We’re still going to track them for our own purposes.” is fine, but it’s a lie to say that the switch turns off location history. The switch should make it clear that all it’s doing is turning off the user facing version and that Google is going to keep tracking you whether you like it or not.
That would be honest, but presumably people wouldn’t like it if they knew, so Google wasn’t truthful about it.
The reporting is true, and Google deserves the consternation they are getting for misleading users.
"Creates a private map of where you go with your signed in devices".
In this case we have invasive surveillance from Chrome and Android which is not a secret and cause of growing concern. As is deceptive communication about 'improving user experience' which is meaningless without specifics and is intentionally used to mislead, and downright fraud with 'location buttons' that do not turn off location.
The fact that you didn't seek to engage these points and immediately sought to question motives betrays an agenda.
Unfortunately these are unethical actions whoever does the research and this kind apologism seeks to derail discussion and distract from the creepy behavior of bad actors.
Google opened the way for collecting massive amounts of data by any means possible (including and not limited to tricking the user into thinking they aren't). And people seem pretty at ease with this, comparatively few are as angered by the revelations these days as they were about MS's relatively harmless telemetry.
So going back, at some point you'll miss the times when you thought MS collecting data about your hardware and drivers was "the evilest evil".
MS and Windows 10 are still far behind Google in the data collection game. Especially since they put that privacy wizard front and center at the first setup.
But as I said, every second you spend defending Google and throwing shade at MS because it feels like the cool thing to do, Google adds another data collection item in their policy, hides yet another setting under 5 layers of options, and tricks you again into thinking that toggling a switch to off actually disables it.
It's not like they did exactly that for 10 years now to the point where they are by far the largest private data collection effort in the history of man kind.
And yes, I'm sure MS will follow suit. Why not? Google seems to be doing just fine.
Luckily, in both cases, free/open source software exists to fix both PC's and (some) cellphones!
Take a look at the privacy settings in your Microsoft account, specifically the activity history. They collect very much the same stuff that Google does, only Google lets you disable the data collection and Microsoft doesn't.
Windows on the PC gives me clear options immediately after installation to disable all that and what's left is the "basic" telemetry, which is very light data collection according to today's standards. At least there's nothing personal there. Also I can very well use the PC without a MS account (so far).
On the mobile side... Is MS still into that? Myself along with probably 5 other people do have a Windows 10 mobile phone but again, the privacy wizard apparently did a great job to disable most of the collection of data. And when I go to Settings > Privacy > Location and I disable it it's disabled.
But yeah, this is the kind of ignorant BS that got all of us here in the first place. As long as there's a critical mass of ignorance on the market the likes of Google know they'll always have someone to sell to so they shove this down everyone's throat. MS makes money by selling you the products. Google gives you everything for free. Not raising any flags?
>MS makes money by selling you the products. Google gives you everything for free. Not raising any flags?
Microsoft makes money in all sorts of ways, among them free and ad funded services like Bing, Skype or LinkedIn.
But yes it does raise flags. That's why - as a paying Office 365 (for Mac) subscriber - I was expecting to have more control over data collection than as a user of ad funded services, not less.
First, MS doesn't exist on the mobile side. This might sound irrelevant until you realize that a phone has the potential to be a much more powerful data pump.
Second, most Windows installations aren't tied to a MS account. As such no data can be linked to it and to you in person.
Third, you see this screen (a few settings are off-screen)? [1] It actually disables most data collection, like location. It doesn't just hide it from the dashboard and have you disable it "again" in 3 more places like Google was shown to do these days. Just by disabling Cortana you get rid of most data collection that can be tied to you.
This is the first thing I do on any Windows machine. Between this and the fact that I don't have to log in with a MS account means that my MS activity dashboard only includes my explicit browser logins and this:
We don’t have any data associated with this Microsoft account at the moment.
Or this:
There’s nothing to see here yet. To add some interests in this category, open Cortana’s Notebook on your device.
[1] https://www.bleepstatic.com/images/news/companies/m/microsof...
And here's the point you are in fact missing or rather denying: Microsoft does collect tons of very personal data as well, not just performance data. You are acknowledging as much when you say that disabling it is the first thing you do on any Windows machine.
Most people will never change the defaults, and Microsoft is pushing extremely hard to make all users of their software log in to a Microsoft account. A lot of Office 365 functionality makes no sense without being logged in. So it is important what privacy settings there are, what the defaults are and what you can or cannot disable.
But I think the reason why we are talking past each other is that I don't use Windows. I don't have the settings in your screenshot. I was comparing the settings in my online Microsoft account to the settings in my Google account.
If your main point is that Google is far more dependent on collecting as much personal data as they possibly can than Microsoft, then I fully agree with that. But that makes it all the more baffling why Microsoft behaves so much like Google when it comes to data collection.
1) Harvesting massive amounts of data from their users computers.
2) Harvesting massive amounts of data from their search engine and ad business.
3) Allowing the Chinese government to harvest all of their users data on their Chinese state run data centers.