We use a similar strategy for our HTML5-based IOS app, however the issue is that it might take a few seconds to realize that an error has occurred and the resource should be served stale, which pretty obnoxiously kills any responsiveness.
The ideal way to make this work is to allow requests to be loaded idempotent and thus you can serve a stale request first and then if a more recent version should come down from the server you can re-load that resource individually (say a feed request to your server or something). This of course requires more than just an HTTP header but it's a cleaner solution provided you can prove that loading of resources is idempotent.
> I believe the current proposal for offline web applications is too complicated, fiddly, and brittle.
This is true, but I feel that HTML5 is the problem.
Given that we can't hope for a quantum leap over HTML for quite some time, I expect that one of the JavaScript frameworks will emerge as a powerful application framework for web apps, offline and online.
cappuccino.org is probably a good first generation example.
There are any real possibility for running pure offline application within the browser? obviously without embedding a webbrowser control on your application.
Another requirement is more reliable browser-side storage.
IndexedDB is too slow (an order of magnitude slower than SQLite) and too high-level (with consequences for those who want to handle index migrations or index multiple values).
WebSQL (SQLite) needs to be supported by Firefox so that the web can have two viable storage options, not just IndexedDB. LocalStorage does not count since it is limited to 10mb.
Furthermore, Chrome needs to offer real permission choices to users who want it. At the moment there is no way for an app to be granted storage quotas unless it registers with the Chrome web store.
All the browsers need to come together on this:
1. SQLite must be in all the browsers (at least until IDB proves itself apart from political support from Mozilla). SQLite is probably the most widely-deployed embedded database and IDB is a far cry away.
2. Users must be able to grant storage quotas to web apps directly (without requiring that they be installed through some web store).
This will only happen if we the developers start to get active on these issues and stop assuming that the W3C committee will get everything right without our involvement.
many apps could get by using a combination of LocalStorage (for storing un-synced data generated by the user offline) and this caching strategy (for storing all of the synced data locally). It's not SQL, but it likely covers a decent amount of use cases.
LocalStorage only gives you 10mb. If your user is offline for a week or two you will run into problems. Further, if your user wants all their data with them how would you explain that they can only have 10mb of it?
Again, it's not SQL that we need, but a reliable storage mechanism that scales to gigabytes of browser-side data. SQLite is a reliable storage mechanism, interface issues and advantages of SQL aside.
Yep, agreed, some apps could easily generate more than 10mb in a couple of weeks. It's definitely an issue.
Users could take all their data with them following this proposal since, provided the Cache-Storage was high enough, data that is pushed up to the online web app could be served back to the client (as json data, a web page, etc.) and stored in the web cache like any other type of asset
A synchronous interface to SQLite would be nice as well - I really didn't enjoy using the current asynchronous interface. (I think this might have been part of the original spec. but hasn't been implemented).
I disagree entirely. Trying to make a cache anything other than a cache is a fatal error. The purpose of a cache is to return exactly what would have been returned if the cache wasn't there, only faster, and even minor deviations from that principle rapidly become problematic. The gulf between disposable data that can be optionally used for speeding things up, and canonical copies of data (even locally canonical ones) is not a difference that should be glossed over into one single subsystem, but a difference that should be highlighted and called out and generally fussed over.
For that matter, the W3C should stop calling this a "cache". The offline applications, as specified, are not functioning like cached values. They are functioning like installed applications. If you can't dispose of a "cache" without a thought beyond "Gosh, my app's performance is going to suffer until this cache is repopulated", you are asking for pain, or, alternately, you don't actually have a cache.
To the extent that you may be able to re-use HTTP mechanisms for detecting changes, that's great, but it should be done carefully and consciously to avoid dependencies between the cache and application storage.
Also, I don't see how this secondary proposal deals with the fact that an application install may involve installing files that the browser has not accessed yet, which seems a fatal error to me.
There's not a fatal error there, afaik. I didn't detail how to deal with un-accessed files because the solution seemed trivial: when your application boots up, make sure it prefetches all required assets (i.e. warms the cache) by making ajax requests to each URL.
I'm not sure about your interpretation of what does and doesn't constitute appropriate use of a web cache, either. Do you have any examples of the kind of problem you're foreseeing?
So to "warm the cache" you're basically going to have a list of files that need to be prefetched. Also, in case the network is offline, you need fallbacks for certain unavailable resources.
So why not have a standard mechanism for this that doesn't clutter the HTTP protocol?
There is a difference between Cache-Control and this offline-cache ... Cache-Control specifies an expiry date (that has to make sense even if you're permanently online, as in you can't put an expiry of 1 month if articles may be updated in far less than that), while the offline cache gets invalidated only when you update the manifest and thus provides an easier to deal with mechanism for the distribution of new versions.
I agree with the above comment that you can't really abstract offline away. Offline is offline. People may stay offline from a couple of hours to a whole month and your app has to still function in that period of time - think of Google Reader - personally I like having favorite articles copied locally when I'm going on vacation, but roaming costs are too much for my data plan and I'm definitely turning the net off. So the app has to cache these articles for an indefinite period, but it has to update as soon as I get online (preferably only things that got updated).
"I'm not sure about your interpretation of what does and doesn't constitute appropriate use of a web cache,"
Not web cache, just plain cache cache. Any kind of cache. Trying to abstract cached content and permanent content with one abstraction is a recipe for disaster, a disaster that leaves you without the proper tools to solve it. It's the sort of disaster that comes up a lot in software engineering where I can not provide a single pithy code example in 20 lines, because they you can always reply with "Well, I'd just write these 25 other lines". The problem is deeper; you've written a semantic confusion into the base primitives of your system and you will pay for it.
It's similar to the problem with RPC; you should not and can not simply sweep the difference between local and network communication under the rug. It works small and is incredibly painful in the large. Unfortunately, I can not seem to find a pithy explanation of why this is true. (Perhaps I should write one.)
An installed application is not simply a cached version of the website. Even the W3C proposal has gone too far down that road, but the solution is not to keep going even farther.
> It's similar to the problem with RPC; you should not and can not simply sweep the difference between local and network communication under the rug. It works small and is incredibly painful in the large. Unfortunately, I can not seem to find a pithy explanation of why this is true. (Perhaps I should write one.)
I would be willing to help you write it if you'd like. I understand network disruption, partial failure, and weak/episodic connectivity very well, including scenarios where nodes in a network are required to remain radio-silent for long periods of time. Command-and-control military-style systems are much more general than the Web.
Related, automatic code distribution is a sort of Holy Grail in programming language design. It is difficult for many reasons. Even data is a huge problem. For example, if the toolbar buttons on Google Docs suddenly die (like they did for our marketing rep a few weeks ago), then the application can become extremely hard to use. Web applications don't handle this well today, and a cache doesn't really address this well, either, because it could be that the new resource invalidates an older cache. In other words, it is a version control and configuration management problem.
You didn't finish my quote, and the the part you cut off is not incidental: "you are asking for pain".
Many things at hardware level are indeed asking for pain. That smart people have labored at length to make them work anyhow doesn't mean it wasn't quite painful. I didn't say it was impossible.
For hardware, it can be a sensible tradeoff. For a generalized web application installation method I don't think it is; it writes into the standard that you must know a lot of very subtle things that your average programmer doesn't even see as issues. This is a bad thing to put into a standard. Better to do something stupider-but-easier, and let people get clever if they think they have some compelling need.
Though stepping one "why" back, the real problem here is that the W3C shouldn't be trying to prospectively write a standard this complicated at all. Why they keep doing that and why people care when they do is beyond me, their track record for creating standards de novo is terrible. This really does call for a few browsers to just try some stuff and see what sticks, no matter how "proprietary" it may make the solution, because no group of people is smart enough to solve all the problems that arise here without even a quarter of an implementation in hand somewhere.
I agree, but am not sure for the exact reasons you state.
Put simply, offline applications need to support partial failure. The author of this article seems to want to write web apps without having to have much concern for partial failure. That is my own summary and take-away. If you agree, then we are on the same page.
Not dealing with partial failure means succombing to Fallacies of Distributed Computing - period. I wouldn't even trust the browser to do atomic swaps of cache', anyway, and I wouldn't even want to think about what that code would look like, or what the spec for the author's ideal would look like.
Bottom line: The author is defining words like "brittle" from one perspective only. What he wants is probably better provided by a library on top of caching. Moreover, I shriek at the idea of an application having to know its "size". That exposes the representation of the resource.
A good early example of handling partial failure is the Zimbra mail & calendar app, the original "AJAX" app on the Internet, before Gmail/Google Calendar even. The Zimbra WebGUI supported an explicit "serverless" mode, which allowed for saving things like outgoing e-mail to a local repository and then, when network availability is detected, re-synchronize with the server and transparently transition to "server" mode.
How does the browser know the set of pages to be cached for a particular offline app, though? Do you have to provide a static sitemap at a known URL? But to support multiple apps deliverable from a single subdomain, you would need to specify the sitemap's URL in some webpage, perhaps as an attribute to some HTML element. So each sitemap would be a sort of Manifest which controls the Cache for an app.
By the way, the app's cache obviously ought to be isolated from other apps' caches, right? Especially if you let the user grant an enlarged cache as a per-app permission. And the user might want to clear the miscellany browser cache without clearing the app cache, so we had better give this Application Cache a distinct name.
That's a decent point about clearing the cache - it should be distinct from flushing assets indicated for offline use. That distinction could be drawn by determining whether or not each asset was served with stale-if-error.
26 comments
[ 2.7 ms ] story [ 64.4 ms ] threadThe ideal way to make this work is to allow requests to be loaded idempotent and thus you can serve a stale request first and then if a more recent version should come down from the server you can re-load that resource individually (say a feed request to your server or something). This of course requires more than just an HTTP header but it's a cleaner solution provided you can prove that loading of resources is idempotent.
This is true, but I feel that HTML5 is the problem.
Given that we can't hope for a quantum leap over HTML for quite some time, I expect that one of the JavaScript frameworks will emerge as a powerful application framework for web apps, offline and online.
cappuccino.org is probably a good first generation example.
IndexedDB is too slow (an order of magnitude slower than SQLite) and too high-level (with consequences for those who want to handle index migrations or index multiple values).
WebSQL (SQLite) needs to be supported by Firefox so that the web can have two viable storage options, not just IndexedDB. LocalStorage does not count since it is limited to 10mb.
Furthermore, Chrome needs to offer real permission choices to users who want it. At the moment there is no way for an app to be granted storage quotas unless it registers with the Chrome web store.
All the browsers need to come together on this:
1. SQLite must be in all the browsers (at least until IDB proves itself apart from political support from Mozilla). SQLite is probably the most widely-deployed embedded database and IDB is a far cry away.
2. Users must be able to grant storage quotas to web apps directly (without requiring that they be installed through some web store).
This will only happen if we the developers start to get active on these issues and stop assuming that the W3C committee will get everything right without our involvement.
Again, it's not SQL that we need, but a reliable storage mechanism that scales to gigabytes of browser-side data. SQLite is a reliable storage mechanism, interface issues and advantages of SQL aside.
Users could take all their data with them following this proposal since, provided the Cache-Storage was high enough, data that is pushed up to the online web app could be served back to the client (as json data, a web page, etc.) and stored in the web cache like any other type of asset
https://groups.google.com/a/chromium.org/group/chromium-html...
For that matter, the W3C should stop calling this a "cache". The offline applications, as specified, are not functioning like cached values. They are functioning like installed applications. If you can't dispose of a "cache" without a thought beyond "Gosh, my app's performance is going to suffer until this cache is repopulated", you are asking for pain, or, alternately, you don't actually have a cache.
To the extent that you may be able to re-use HTTP mechanisms for detecting changes, that's great, but it should be done carefully and consciously to avoid dependencies between the cache and application storage.
Also, I don't see how this secondary proposal deals with the fact that an application install may involve installing files that the browser has not accessed yet, which seems a fatal error to me.
I'm not sure about your interpretation of what does and doesn't constitute appropriate use of a web cache, either. Do you have any examples of the kind of problem you're foreseeing?
So why not have a standard mechanism for this that doesn't clutter the HTTP protocol?
There is a difference between Cache-Control and this offline-cache ... Cache-Control specifies an expiry date (that has to make sense even if you're permanently online, as in you can't put an expiry of 1 month if articles may be updated in far less than that), while the offline cache gets invalidated only when you update the manifest and thus provides an easier to deal with mechanism for the distribution of new versions.
I agree with the above comment that you can't really abstract offline away. Offline is offline. People may stay offline from a couple of hours to a whole month and your app has to still function in that period of time - think of Google Reader - personally I like having favorite articles copied locally when I'm going on vacation, but roaming costs are too much for my data plan and I'm definitely turning the net off. So the app has to cache these articles for an indefinite period, but it has to update as soon as I get online (preferably only things that got updated).
I don't think this does clutter the HTTP, in fact the proposal even provides an alternative to the one single addition it does make to HTTP.
I'm not clear on the other points you've made, sorry
Not web cache, just plain cache cache. Any kind of cache. Trying to abstract cached content and permanent content with one abstraction is a recipe for disaster, a disaster that leaves you without the proper tools to solve it. It's the sort of disaster that comes up a lot in software engineering where I can not provide a single pithy code example in 20 lines, because they you can always reply with "Well, I'd just write these 25 other lines". The problem is deeper; you've written a semantic confusion into the base primitives of your system and you will pay for it.
It's similar to the problem with RPC; you should not and can not simply sweep the difference between local and network communication under the rug. It works small and is incredibly painful in the large. Unfortunately, I can not seem to find a pithy explanation of why this is true. (Perhaps I should write one.)
An installed application is not simply a cached version of the website. Even the W3C proposal has gone too far down that road, but the solution is not to keep going even farther.
I would be willing to help you write it if you'd like. I understand network disruption, partial failure, and weak/episodic connectivity very well, including scenarios where nodes in a network are required to remain radio-silent for long periods of time. Command-and-control military-style systems are much more general than the Web.
Related, automatic code distribution is a sort of Holy Grail in programming language design. It is difficult for many reasons. Even data is a huge problem. For example, if the toolbar buttons on Google Docs suddenly die (like they did for our marketing rep a few weeks ago), then the application can become extremely hard to use. Web applications don't handle this well today, and a cache doesn't really address this well, either, because it could be that the new resource invalidates an older cache. In other words, it is a version control and configuration management problem.
> If you can't dispose of a "cache" without a thought beyond "Gosh, my app's performance is going to suffer until this cache is repopulated"
According to what you're saying: "Write cache" or "dirty cache" are oxymorons, and yet these terms are used everywhere.
Many things at hardware level are indeed asking for pain. That smart people have labored at length to make them work anyhow doesn't mean it wasn't quite painful. I didn't say it was impossible.
For hardware, it can be a sensible tradeoff. For a generalized web application installation method I don't think it is; it writes into the standard that you must know a lot of very subtle things that your average programmer doesn't even see as issues. This is a bad thing to put into a standard. Better to do something stupider-but-easier, and let people get clever if they think they have some compelling need.
Though stepping one "why" back, the real problem here is that the W3C shouldn't be trying to prospectively write a standard this complicated at all. Why they keep doing that and why people care when they do is beyond me, their track record for creating standards de novo is terrible. This really does call for a few browsers to just try some stuff and see what sticks, no matter how "proprietary" it may make the solution, because no group of people is smart enough to solve all the problems that arise here without even a quarter of an implementation in hand somewhere.
Storage systems have write caching.
It is a pretty important mechanism, and may or may not be painful depending on reliability requirements.
Put simply, offline applications need to support partial failure. The author of this article seems to want to write web apps without having to have much concern for partial failure. That is my own summary and take-away. If you agree, then we are on the same page.
Not dealing with partial failure means succombing to Fallacies of Distributed Computing - period. I wouldn't even trust the browser to do atomic swaps of cache', anyway, and I wouldn't even want to think about what that code would look like, or what the spec for the author's ideal would look like.
Bottom line: The author is defining words like "brittle" from one perspective only. What he wants is probably better provided by a library on top of caching. Moreover, I shriek at the idea of an application having to know its "size". That exposes the representation of the resource.
By the way, the app's cache obviously ought to be isolated from other apps' caches, right? Especially if you let the user grant an enlarged cache as a per-app permission. And the user might want to clear the miscellany browser cache without clearing the app cache, so we had better give this Application Cache a distinct name.