My lament with Facebook has always been the corraling of the web into a private network instead of the privacy that has stayed in the news.
Apps for different things has made things worse. Today rarely a new business open a website they just post their details on Facebook or an app specific to the industry. And we have to install an app or login to Facebook to access the data.
Google and twitter helping publishers might be good for publishers and the open web but I dont think its enough to save it.
Maybe they are talking about using the HTML AppCache/manifest and a script or something that downloads all of the resources in the manifest and background-preloads them in the Google/Facebook/Twitter etc. page/app. Maybe you don't need a script, just a frame. Maybe a special 'article.html' resource or something that is not allowed to have ads/popups/JS etc.
As long as they don't limit that to certain favored organizations.
Yeah, I'm not sure I trust Google so much anymore and I trust Facebook even less. It'd have to be a W3 group somehow with all the players involved. Definitely some of them would play spoiler like Microsoft did back in the day. Apple would probably play spoiler with this, they like their App Store it makes them money.
Fair point but the real solution would be to do those things all fast from the one site somehow. Not many url's many resources etc. How to do that I dunno.
You are not entitled to ad revenue or client-side analytics. The fact they work at all is just an odd combination of historical missteps that will be corrected in time.
That is, if we can ever get people to stop using Chrome. Chrome is Google's Embrace, Extend, Extinguish. Release open browser, add features, then restrict features like adding a default embedded "YouTube App" inside Chrome to get around ad blockers. Add default "Google Apps" apps to make Google services better than any other service you can use, because Chrome secretly loaded native pre-cached client-side, not web, apps on your computer.
It turns out your default content consumption applications shouldn't be written by corporations intertwined with advertising revenue.
It'll take the US government another 40 years to re-define monopoly scenarios capable of handling current abuses of power. In the meantime, sit back and enjoy the land grabs, rising stock prices, and shuttered startups who can't compete with bundled platforms.
Can someone who works on web development for a major publisher please try and explain why such sites take so long to load?
Just by opening devtools in Chrome I can tell that a lot of sites aren't bothering to minify HTML, aren't bothering to asynchronously load JavaScript, include more frameworks and widgets than anyone could possibly use and don't bother to cache anything (or cache very little). And with all the pointless junk they include, they still can't use proper HTML tags and attributes nor include some basic accessibility features.
I really hate having to reload pages because something didn't load properly or having to turn off JavaScript and/or images just because a site is loading a huge file that is blocking the display of a paragraphs of text. I know some people like to use something like Lynx or only allow HTML to be loaded - but my web browsing shouldn't have to become sadistic.
Making Google, Twitter or Facebook load news articles is a terrible alternative. Given the wide array of compression tools, caching (including service workers on >40% of browsers, appcache on >90% and pretty much everyone having HTTP caching) and the sheer amount of information on improving website performance that's out there, making news websites load fast really should be a solved problem that doesn't need Google to solve it.
> Can someone who works on web development for a major publisher please try and explain why such sites take so long to load?
1. The ad/tracking industry seems to be incapable of writing JavaScript that is not awful. I don't have insight why that happens, but all the scripts I had to deal with were layers upon layers of document.write, iframes, redirects and copy&pasted JS from the Netscape 4 era. Even when somebody tries to fix it, the best they can do is to add one more layer of scripts on top of the rotting pile.
2. Business deals are made with no consideration of 3rd party's code quality and performance impact. Once the deal is signed the developers are just tasked with integrating all of the crap the 3rd party vendor wants. It's impossible to object: the vendor is not going to rewrite all of their shitty scripts, especially after the deal is already in place. Even when it's really bad, arguing about it just results in endless meetings, conference calls and at best the 3rd party vendor saying they'll think about putting some little tweak in the backlog for some future release.
And here's the problem with people armchairing publishing, you think the publishers need convincing that they need _faster ads_.
It's not the publishers! They all know our sites are slow, but like pornel said they don't have a say in how efficient their partners/advertisers code is. And they aren't going to rip that shit out and go broke just because people complain.
Ad block and the browsers sunsetting Flash is slowly forcing the ad tech people to change, but it's not near fast enough. In the meantime publishers revenue suffers and people continue to blame _them_ for the experience.
The alternative is they don't do display ads at all and go down the BuzzFeed route, where advertising is indistinguishable from actual content.
If you think it's easy to fix with technology, you are more than welcome to try.
Sorry, I didn't mean to paint such a bleak picture. There are good and smart people working on these sites, and some newspapers have technically very good code (apart from ads :)
There are also lots of very hard constraints. The publishing process is complex and on strict schedules, especially if it's connected to printing of a daily newspaper. There are hundreds of people involved, so upgrading CMS for a newspaper is like changing wheels of a car while it's moving.
And on top of that monetization on the web is hard. Users hate paywalls. Nobody wants a mobile app, especially not for overpriced in-app payments. But something needs to bring money, and even getting ad inventory that's higher class than "doctors hate this!!!" is also a very hard work.
> Can someone who works on web development for a major publisher please try and explain why such sites take so long to load?
Well, the same problems that beset big enterprise are also found in "big publishing" which is even more severely amplified by the need to have everything live yesterday.
> Given the wide array of compression tools, caching (including service workers on >40% of browsers, appcache on >90% and pretty much everyone having HTTP caching) and the sheer amount of information on improving website performance that's out there, making news websites load fast really should be a solved problem that doesn't need Google to solve it.
Correct, but you are assuming that most developers are a) interested in learning these things b) possess even the fundamental of CS knowledge and skills. By reading HN, I would assume that both a and b apply to the majority here, however, in my own cynical outlook, most "developers" are really just out there winging it and cannot be bothered to learn more than what's provided on w3schools.com.
I'm the co-founder/CTO of an analytics company that serves publishers. My company's data on Facebook vs Google referrals was cited in this NYT article, in fact.
We offer analytics with a similar business model to companies like MixPanel and NewRelic -- that is, not analytics for ad serving and buying, but for improving user experience, user engagement/loyalty, and informing content strategy. Like those companies, we charge money for these analytics -- because they are valuable to the site owners, are hard to calculate, and help them improve their sites over time.
I'll try to summarize what I see happening:
When I first entered this industry (~5 years ago) and launched our first JavaScript integration, the state of JS in the industry was poor. Ad tech companies embedded clunky iframes that would sometimes load entire web stacks in them. Loading a single publisher website was often like loading 10 websites in parallel in a single tab. Many of these ad tech vendors haven't upgraded, so this kind of stuff is still floating around.
I made a focused effort to make a slim JavaScript integration to collect the data we needed for useful reporting to the site owners, using as few bytes as possible. That meant writing all the JS code "from scratch" (no dependencies), forcing async-loading, and beaconing back the minimum needed data.
Over time, we've slimmed down our integration considerably as more engineers have come on-board and came up with ideas to make our JS have less and less impact on page load performance. We've kicked tires on various code minifiers to find the right one, refactored code to simplify it, etc. But my company is rare in this respect. Most vendors (many of them ad tech) simply don't care. Once they integrate their crappy code on the site, it's frozen forever, and they don't touch it.
As one small example of this, my company's analytics system collects data about pages via a backend web crawler that integrates the Schema.org JSON-LD web standard, the same standard that the search engines use. What does this mean? Rather than beaconing back ~50kb of article metadata on every page view of the article (which is what most page analytics vendors do, e.g. Adobe Omniture or Webtrends), we beacon back 1kb of event data on every page view, then crawl the other 50kb from our backend servers. In this way, we don't punish every user, over and over again, for the data we need to collect. This requires us to do a streaming join of these data sets on the backend, which is complicated. But it's complicated for us, the engineers building the analytics system, not for you, the typical web visitor. That's The Way It Should Be.
There is also a shady thing going on where many "data brokers" offer publishers a single JavaScript integration, which starts off slim. But as the publisher's data gets "sold" to other data brokers and ad exchanges, the JavaScript integration gets fatter, because it's now "JS embedding JS embedding JS". You get another (recur) of the JS embed for each additional bizdev deal struck between any two data brokers.
As a company, we have taken a firm stance that we are not an ad tech company. We charge our customers money every month for our services (reporting dashboards and APIs), and thus we don't need to "monetize on the side".
But you have no idea how against the grain this is in the media industry. On the web, "monetize on the side" is the primary business model. In fact, the CEO of a major media site once told me, "My only problem with your service is that you charge money for it. I don't think we should pay for services. Everything should be free, and you guys should figure out how to make money on your own."
With bad actors among the ad tech industry popping up every day who are happy to take advantage of this mindset, it's no surprise that the page load situation has gotten out of control.
"People often favor mobile apps because they are faster, cleanly formatted and are constantly updated to take advantage of the evolving features of new smartphones."
That's all conjecture. The truth is that the reasons people will use an App over a website is subjective. Whatever reasons people have should be gathered from research and studies and not just theories based off of a narrative from site owners.
I still use the YouTube app but not the Twitter app. I prefer to use Google's site over the widget/app but will sometimes on the spur of the moment.
21 comments
[ 4.7 ms ] story [ 59.8 ms ] threadAs long as they don't limit that to certain favored organizations.
That is, if we can ever get people to stop using Chrome. Chrome is Google's Embrace, Extend, Extinguish. Release open browser, add features, then restrict features like adding a default embedded "YouTube App" inside Chrome to get around ad blockers. Add default "Google Apps" apps to make Google services better than any other service you can use, because Chrome secretly loaded native pre-cached client-side, not web, apps on your computer.
It turns out your default content consumption applications shouldn't be written by corporations intertwined with advertising revenue.
It'll take the US government another 40 years to re-define monopoly scenarios capable of handling current abuses of power. In the meantime, sit back and enjoy the land grabs, rising stock prices, and shuttered startups who can't compete with bundled platforms.
Just by opening devtools in Chrome I can tell that a lot of sites aren't bothering to minify HTML, aren't bothering to asynchronously load JavaScript, include more frameworks and widgets than anyone could possibly use and don't bother to cache anything (or cache very little). And with all the pointless junk they include, they still can't use proper HTML tags and attributes nor include some basic accessibility features.
I really hate having to reload pages because something didn't load properly or having to turn off JavaScript and/or images just because a site is loading a huge file that is blocking the display of a paragraphs of text. I know some people like to use something like Lynx or only allow HTML to be loaded - but my web browsing shouldn't have to become sadistic.
Making Google, Twitter or Facebook load news articles is a terrible alternative. Given the wide array of compression tools, caching (including service workers on >40% of browsers, appcache on >90% and pretty much everyone having HTTP caching) and the sheer amount of information on improving website performance that's out there, making news websites load fast really should be a solved problem that doesn't need Google to solve it.
1. The ad/tracking industry seems to be incapable of writing JavaScript that is not awful. I don't have insight why that happens, but all the scripts I had to deal with were layers upon layers of document.write, iframes, redirects and copy&pasted JS from the Netscape 4 era. Even when somebody tries to fix it, the best they can do is to add one more layer of scripts on top of the rotting pile.
2. Business deals are made with no consideration of 3rd party's code quality and performance impact. Once the deal is signed the developers are just tasked with integrating all of the crap the 3rd party vendor wants. It's impossible to object: the vendor is not going to rewrite all of their shitty scripts, especially after the deal is already in place. Even when it's really bad, arguing about it just results in endless meetings, conference calls and at best the 3rd party vendor saying they'll think about putting some little tweak in the backlog for some future release.
It's not the publishers! They all know our sites are slow, but like pornel said they don't have a say in how efficient their partners/advertisers code is. And they aren't going to rip that shit out and go broke just because people complain.
Ad block and the browsers sunsetting Flash is slowly forcing the ad tech people to change, but it's not near fast enough. In the meantime publishers revenue suffers and people continue to blame _them_ for the experience.
The alternative is they don't do display ads at all and go down the BuzzFeed route, where advertising is indistinguishable from actual content.
If you think it's easy to fix with technology, you are more than welcome to try.
There are also lots of very hard constraints. The publishing process is complex and on strict schedules, especially if it's connected to printing of a daily newspaper. There are hundreds of people involved, so upgrading CMS for a newspaper is like changing wheels of a car while it's moving.
And on top of that monetization on the web is hard. Users hate paywalls. Nobody wants a mobile app, especially not for overpriced in-app payments. But something needs to bring money, and even getting ad inventory that's higher class than "doctors hate this!!!" is also a very hard work.
Well, the same problems that beset big enterprise are also found in "big publishing" which is even more severely amplified by the need to have everything live yesterday.
> Given the wide array of compression tools, caching (including service workers on >40% of browsers, appcache on >90% and pretty much everyone having HTTP caching) and the sheer amount of information on improving website performance that's out there, making news websites load fast really should be a solved problem that doesn't need Google to solve it.
Correct, but you are assuming that most developers are a) interested in learning these things b) possess even the fundamental of CS knowledge and skills. By reading HN, I would assume that both a and b apply to the majority here, however, in my own cynical outlook, most "developers" are really just out there winging it and cannot be bothered to learn more than what's provided on w3schools.com.
We offer analytics with a similar business model to companies like MixPanel and NewRelic -- that is, not analytics for ad serving and buying, but for improving user experience, user engagement/loyalty, and informing content strategy. Like those companies, we charge money for these analytics -- because they are valuable to the site owners, are hard to calculate, and help them improve their sites over time.
I'll try to summarize what I see happening:
When I first entered this industry (~5 years ago) and launched our first JavaScript integration, the state of JS in the industry was poor. Ad tech companies embedded clunky iframes that would sometimes load entire web stacks in them. Loading a single publisher website was often like loading 10 websites in parallel in a single tab. Many of these ad tech vendors haven't upgraded, so this kind of stuff is still floating around.
I made a focused effort to make a slim JavaScript integration to collect the data we needed for useful reporting to the site owners, using as few bytes as possible. That meant writing all the JS code "from scratch" (no dependencies), forcing async-loading, and beaconing back the minimum needed data.
Over time, we've slimmed down our integration considerably as more engineers have come on-board and came up with ideas to make our JS have less and less impact on page load performance. We've kicked tires on various code minifiers to find the right one, refactored code to simplify it, etc. But my company is rare in this respect. Most vendors (many of them ad tech) simply don't care. Once they integrate their crappy code on the site, it's frozen forever, and they don't touch it.
As one small example of this, my company's analytics system collects data about pages via a backend web crawler that integrates the Schema.org JSON-LD web standard, the same standard that the search engines use. What does this mean? Rather than beaconing back ~50kb of article metadata on every page view of the article (which is what most page analytics vendors do, e.g. Adobe Omniture or Webtrends), we beacon back 1kb of event data on every page view, then crawl the other 50kb from our backend servers. In this way, we don't punish every user, over and over again, for the data we need to collect. This requires us to do a streaming join of these data sets on the backend, which is complicated. But it's complicated for us, the engineers building the analytics system, not for you, the typical web visitor. That's The Way It Should Be.
There is also a shady thing going on where many "data brokers" offer publishers a single JavaScript integration, which starts off slim. But as the publisher's data gets "sold" to other data brokers and ad exchanges, the JavaScript integration gets fatter, because it's now "JS embedding JS embedding JS". You get another (recur) of the JS embed for each additional bizdev deal struck between any two data brokers.
As a company, we have taken a firm stance that we are not an ad tech company. We charge our customers money every month for our services (reporting dashboards and APIs), and thus we don't need to "monetize on the side".
But you have no idea how against the grain this is in the media industry. On the web, "monetize on the side" is the primary business model. In fact, the CEO of a major media site once told me, "My only problem with your service is that you charge money for it. I don't think we should pay for services. Everything should be free, and you guys should figure out how to make money on your own."
With bad actors among the ad tech industry popping up every day who are happy to take advantage of this mindset, it's no surprise that the page load situation has gotten out of control.
By the way, what's your company called?
That's all conjecture. The truth is that the reasons people will use an App over a website is subjective. Whatever reasons people have should be gathered from research and studies and not just theories based off of a narrative from site owners.
I still use the YouTube app but not the Twitter app. I prefer to use Google's site over the widget/app but will sometimes on the spur of the moment.