Here is a question I hoped to be answered, but it wasn’t: Why doesn’t the HTTP/2 protocol use SRV DNS records¹? Using it would solve some real problems:
1. The well-known resilience of mail can be largely attributed to its use of MX records to specify the use of backup servers and/or load balancing. SRV records are essentially a generalization of MX records for any service, not just mail, so using SRV records would bring this benefit to any protocol which uses them.
2. The problem of “www” or no “www” would be entirely eliminated; no need for non-standard so-called ANAME or ALIAS “records”. Just imagine – if, in an alternate universe, mail servers hadn’t used MX records, it would then have been necessary to have an A record on the bare domain name, and later, when the Web was created in this alternate universe, the web server would have to be the same server as the mail server, since both now would use the bare domain name. In our universe, this position is now solely occupied by the HTTP protocol, which causes lots of pain in the DNS when redundant (and harder to update) A records need to be put on the bare domain name just for HTTP. Not to mention IPv6, which, for HTTP to work, requires an additional AAAA record on the bare domain.
3. The “everything is on port 80, let’s reinvent everything on top of HTTP” problem could certainly have been entirely avoided, and could be slowly fixed, by using SRV records. (A server concerned about clients needing to traverse old-style restrictive firewalls could provide a low-priority port 80 server as a fallback.)
I've recently done some work using SRV (for Minecraft, which uses it).
I agree it's a pretty decent system, and something I'd like to see more systems adopt it.
Interestingly, I can't actually think of a reason why a (customised) web browser couldn't use it now. If a browser understood SRV records it could use the correct port. There might be some odd interactions with the HTTP 1.1 Host header, but I haven't thought about it much.
Of course, browsers that don't understand it wouldn't be able to follow links that use it which is something of a downside I guess.
According to the RFC for SRV records, applications SHOULD NOT use SRV records unless explicitly allowed in the standard/RFC for that protocol. There is no mention of SRV records in either the old HTTP RFC or the new HTTP/2 draft RFC; therefore, clients are not permitted to use SRV records.
Indeed, since the HTTP/1.1 RFC was issued in 1999 and the SRV RFC in 2000 there is no way it could have mentioned it.
If there are technical reasons why it can't work that's one thing. If some verbiage in the RFC is all that holds it back it's easy to fix: try it out, then issue a new RFC.
And an RFC is not a standard, and all life is meaningless? No, a SHOULD NOT should be taken seriously, not ignored.
The SRV RFC included that language for a reason, and probably had HTTP in mind when they wrote it, and therefore wanted HTTP clients not to use SRV records randomly. Since then, HTTP has has plenty of time to add SRV usage in its minor revisions, but has chosen not to. But now, with HTTP/2, a major revision, it is the perfect time to add this kind of thing.
I'm sorry - I just read that other thread you linked to. Others made the same point to you then, and you still seem stuck on that point.
RFCs are not like ISO standards. The name says it all: "request for comments".
RFCs reflect the ideal of "rough consensus and running code".
The SRV RFC was written by an author from Microsoft aimed at domain controller discovery. They probably didn't think of HTTP at all.
There's an expired draft RFC for SRV+HTTP[1].
You are right: SHOULD NOT should be taken seriously. But if you are (say) Mozilla, Microsoft or Google and writing a webbrowser you probably know enough to understand when you can break a RFC.
Different RFCs have different levels. RFC 2782 is a “Standards Track” RFC and a “Proposed Standard”, and should be taken seriously indeed (there is only one “higher” level).
The SRV RFC was not by Microsoft. It has three authors, the other two being one Arnt Gulbrandsen from Troll Tech (of the Qt library fame) and Paul Vixie of the Internet Software Consortium, who is also the author of BIND. I actually am inclined to think that they did consider HTTP seriously.
SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that
there may exist valid reasons in particular circumstances when the
particular behavior is acceptable or even useful, but the full
implications should be understood and the case carefully weighed
before implementing any behavior described with this label.
> Why doesn't the HTTP/2 protocol use SRV DNS records?
This was discussed by the WG. The best rationale is given in the thread starting from this mail[0] from Will Chan at Google, followed up by this one[1] from Amos Jeffries (Squid Proxy). The summary is: waiting for SRV lookups can be slower than just doing straight-up A records, so no client in a competitive space is going to do it.
Everyone agrees that in an ideal world this would be great, but in practice no-one is going to be the first person to move on that front. The wins aren't big enough and the cost is too high.
Reading that I'm not sure the problem is that SRV lookups are slower, but that forcing clients to do a SRV lookup, wait, and then a A/AAAA queries is slower:
If I own a popular client and I switch to mandatory SRV lookup, and don't issue A and AAAA queries until the SRV lookup fails, then
for the services and paths where SRV lookup doesn't work, user experience
will be vastly degraded.[1]
However,it appears that making SRV optional remains a possibility:
Note that none of what I've said necessarily applies to optionally doing
SRV and using it when the results are available.
No, on the contrary, the SRV standard specifies that clients SHOULD NOT use SRV records for protocols for which the standards do not explicitly allow for SRV records.
Multi-request resolving (QDCOUNT > 1) would solve that problem, which is a lesser known feature of the DNS protocol. Clients could ask for all A, AAAA, and SRV for a given domain, and get all the replies in a single round-trip.
Few if any resolvers support this feature, but it show that the problem we have is because of legacy assumption that are no longer true. Fix those, and the problem goes away.
The key words in your quote is “for the services and paths where SRV lookup doesn't work”. You get the same problem today where A lookup doesn’t work. This is an abnormal situation. As I wrote in another comment, normally you get an answer immediately (possibly an empty one where no SRV record exists). I don’t know why a situation with no reply should be considered common enough to optimize for. (Clients which really wanted to mitigate this could certainly detect this case when it happens and keep a blacklist.)
I am not convinced. The same argument (“another round-trip makes it slow”) is sometimes made about Javascript libraries, and the corresponding counter-argument applies here, too: It’s usually cached, anyway, and DNS caches are also much better, since DNS record caching is much more deeply integrated in your OS, your ISP, etc, than HTTP caching, which is usually only in each individual HTTP client. The cache times are also longer for DNS (or could certainly be so, since with SRV handling redundancy there is no need to keep low TTLs anymore).
Notice that Amos was arguing for a happy-eyeballs approach, which removes the round trip. The issue isn't the wait for an extra round trip, it's making it mandatory to wait for an SRV result before using an A result.
If the language says "You MUST consult SRV records for _http2._tcp before you use the result of an A lookup", then anything that delays that SRV lookup will delay page resolution. Browsers are strongly disincentivised from making anything slow, so it's just safer not to add the extra delay point. If we specced that, no-one would do it.
Better to have what we have now, which is: do SRV if you want.
The SRV record should normally be cached, so it would not be slow just for existing.
What Amos was talking about is the problem of some resolvers not returning an answer at all, not even NXDOMAIN or an empty authoritative answer, due to the fact that he queried for AAAA records and some resolvers are buggy and can’t handle those queries. SRV is irrelevant to this problem, except that using “happy-eyeballs” and requiring A and AAAA has problems.
What William Chan (陈智昌) was talking about was where a query for SRV records fail, causing a long timeout. But this is not the normal case; normally you get an answer (perhaps an empty one) immediately. And since this has nothing to do with IPv6 and buggy resolvers, I don’t know why this situation should be considered common enough to optimize for. (Clients which really wanted to mitigate this could certainly detect this case when it happens and keep a blacklist.)
> […] anything that delays that SRV lookup will delay page resolution.
You keep implying that such things will happen. They won’t. There is no reason for an SRV record lookup to delay page resolution any more than an A (or AAAA) record lookup.
> If we specced that, no-one would do it.
Are you seriously saying that even if the HTTP/2 standard came out and it said “A client MUST lookup an SRV record and use it if one exists, and not use any A or AAAA records before the non-existence of an SRV record can be confirmed.”, that the browser vendors would simply ignore this? If that’s the case, why do we even have a standards process? Why do we even pretend to write RFCs if we can just assume that every implementer will ignore the standard and do whatever they feel like?
No, a thing which would be a net win for the Internet in general should go in the standard. RFC writers should not bow to fears of what Google, IBM, or General Motors might or might not do. Or, at least, wait until they make the threat of non-compliance openly.
Last I looked, the HTTP/2 RFC didn't even specify DNS at all.
I regard this as the worst-case scenario. It is a baffling omission. The HTTP URI scheme [RFC7230] does nothing to clarify matters:
"If host is a registered name, the registered name is an indirect identifier for use with a name resolution service, such as DNS..."
In my view the WG has the opportunity to definitively fix the problems of dual-stack discovery and of apex record service names, and the refusal to do so is exceedingly disappointing. We're left with implementations of Happy Eyeballs that have lookup race conditions and therefore nondeterministic behaviour.
All the arguments against using service records are actually complaints about artefacts arising from the current ambiguity in resolution practices, and thereby perpetuate the problem.
The proper choice should be that all IPv6 sites should be resolved via SRV rather than raw AAAA lookup - this is the soft introduction that over the years will ramp up to a global first preference for service records, and remains compatible as a variation on Happy Eyeballs.
But entrenched positions don't change and I have no clout with the WG or with browser product managers, so I gave up.
I'm using nginx with SPDY and I've noticed that if I start downloading a file, I can no longer load pages on the website. I get an error message in Firefox saying "The connection was interrupted". If I do a ctrl+F5 then I can continue browsing the site. Does this have to do with the "one TCP connection"?
I see this behavior too on SPDY-using sites, including ones on CloudFlare. Everyone is constantly going on about the benefits of parallel requests that SPDY / HTTP/2 brings, but I'm not seeing it. Can anyone explain? I'm about to go back to domain-sharding resources to solve this for some of my work even though CloudFlare says that it's not recommended with SPDY.
It should not. That sounds like a bug or configuration problem somewhere.
SPDY and HTTP/2 can multiplex their data streams across a single connection. That is to say, they can receive 2+ datastreams on the same TCP connection at the same time.
i'm a firefox dev. That sounds like a sender (i.e. server) bug, but I'd like to rule out a client problem. Can you send me a specific steps-to-reproduce at pmcmanus@mozilla.com ?
I think I may have just ruled out a client problem, I just tried it in Chrome and it has the same behavior. Are the specific steps still useful or should I follow up with the nginx folks instead?
Under the "Why Revise HTTP"
http://http2.github.io/faq/#why-revise-http section the summary is basically "So we don't have to be as mindful of the number of HTTP requests" Is this really a compelling reason? I would argue it is not, especially if we have to trade a human-readable and easy to troubleshoot text-based protocol with a binary one. It seems like we are solving a small problem by creating a larger one.
The correct answer is that HTTP/2 has to do this because Google is unable to show you a simple search box without making over a dozen HTTP requests.
Their more involved properties require dozens to even hundreds of requests in order to perform complex tasks like showing a list of plain text posts.
And, of course, they can't possibly impose any kind of discipline on the way they do things, so instead the entire internet has to change to accommodate them.
It's so easy to blame everything on Google/Microsoft/$OTHER_BLACK_SHEEP being stupid. I don't know how many websites you've been to, but a large number of them perform hundreds of requests, not just Google. The modern web is full of images, CSS, webfonts, dynamically loaded Javascript, mashups and other things, each which require one or more HTTP requests. Having lots of requests has been the reality for a large part of the web for a number of years now.
Now I could optimize that even more by inlining everything with data-inline: but that would impact caching code and be strictly worse for the network.
A property from google even websearch has fancy transitions and animations and dynamic content. And Google has every incentive to limit the number of connections to their page. When I worked there I worked on some of those technologies specifically to avoid too many connections.
Blaming Google for not being disciplined enough disregards their economic incentives to be disciplined enough. They invested a lot of energy in this before SPDY came out. There comes a time where you just have to recognize that the protocol is fighting against you and either give up or evolve the protocol.
Google created GWT which uses every trick under the sun to reduce connection counts. I use it on my sites and if you don't let your web property devolve into a bolt-on JS mire (as many large sites do, i.e. Borders recently made 248 requests to load), you can load the page/app in 4 requests.
(1) index.html,
(2) app.nocache.js
(3) A56284CFC0D680C7DBA0F93CA9AE8F75.cache.js (cache fragment for given page/state)
(4) XHR for page data/state to display
Now, if you've got some CSS and minor images/fonts, these can be an your options request (5) resource bundle. On many Google properties, (2 and 5) are inlined into (1).
The take away? These technologies and frameworks have been out there for a long time (8 years in the case of GWT), so you don't have to make hundreds of requests if you're disciplined. I can only assume Google has thrown in the towel trying to get script kiddies to ever be disciplined.
Exactly right. Humans only exist to serve their tools.
No, wait a sec. That doesn't sound right. Could it be that our tools exist to serve humans? And that rather than putting effort into accommodating ourselves to the limitations of our current tools, we should instead put our effort into making tools that work the way we want?
Nah, sounds crazy. If HTTP/1 was good enough for my grand-dad, it's good enough for me.
Human-readable is only a matter of tooling. Why is HTTP human-readable? Because you have a lot of tools that understand ASCII. Without them, HTTP is just as unreadable to you as any other stream of bytes.
The tools don't exist right now, but they will become available. Existence of tooling follows demand. HTTP 2 has a large momentum so tooling will exist.
When I'm debugging HTTP, 98% of the time it's either with curl, Wireshark, or by dumping state in my own server-side code. Things like curl will work just the same. Wireshark has an HTTP/2 plugin; given their history, I'm sure it'll be excellent by the time there's much HTTP2 in the wild. And anybody writing server-side code is hacking together their own tools anyhow.
The rare occasion where this will be a big difference for me is telnetting to port 80 and typing HTTP by hand. Honestly, I only do that out of habit; I can do the same things just as well with curl. But even when HTTP/2 ships, I'm sure it will be at least a decade before servers stop supporting HTTP/1.1. Probably longer. So I'm not expecting this to make a big difference to my debugging habits.
I don't mind the binary protocol that much, but the apparent assumption that every endpoint is a fat stateful cache (push, multiplexing) is madness.
Have they actually worked out how to implement a HTTP/2 front end cache in the face of ill-behaved clients, crashy servers, and the way wireless breaks a lot of the assumptions that would make "one big TCP session" a good idea in the first place?
If TCP+TLS handshakes are so expensive, wouldn't it be a better investment of everyone's resources to work on an L3 protocol for setting up a fast second session? Leave the multiplexing to packet people.
> I don't mind the binary protocol that much, but the apparent assumption that every endpoint is a fat stateful cache (push, multiplexing) is madness.
A client can say it does not want push in the settings frame up front. And then it can make one request at a time, effectively avoiding multiplexing, although each frame in the response will have a 9-byte frame header (similar to the overhead in chunking today in 1.x).
By saying it does not want push, it relieves itself of having an object cache. It will still be on the hook for having an HPACK table as state. (In practice this will be coded once in each language in a library and most developers will never think about it.)
> Have they actually worked out how to implement a HTTP/2 front end cache in the face of ill-behaved clients, crashy servers, and the way wireless breaks a lot of the assumptions that would make "one big TCP session" a good idea in the first place?
> If TCP+TLS handshakes are so expensive, wouldn't it be a better investment of everyone's resources to work on an L3 protocol for setting up a fast second session? Leave the multiplexing to packet people.
It's multiplexing with declared weighting and dependency trees. So for example, the client can tell the server that it wants files A, B, C, and D, but to have B, C, D is useless until it has A, and once A is finished, the relative importance of B is twice as much as C or D. That's much easier to do on one socket.
Also one socket gets the header compression in HPACK to benefit from all the headers going through the same state engine. Giant sets of cookies (more common than we'd like to believe) are now more efficiently expressed.
One socket also helps the server-side keep statistics and logs more organized as related requests are together.
I understand your suggestion to do it on a lower level, and that's a fine idea, but I also understand why the working group did it on an HTTP layer, especially with SPDY as precedent.
I work at Akamai, and I'm one of the open source library contributors (the Common Lisp one). These opinions are my own.
Granted, I think it was overall a poor decision to introduce that scheme prefix as well. It's just that, many arguments against a new scheme prefix also applied in that case, and many arguments for that new scheme prefix would apply in this case.
Because that would break every hyperlink on the web, and would create a separate web with content that existing clients or intermediaries would be excluded from. Much more evolvable to keep the wire protocol version orthogonal to URI.
Further, technically speaking, the URL scheme has little to with the protocol used , it is merely a way of linking a hierarchical namespace to a spec (or more) that tells a developer how to resolve it. That means we have a lot of flexibility in terms of protocol (we don't even need a protocol defined in some cases - mailto: URIs for example).
The only way an upgrade this big will happen is having a single server that speaks both, one that's a drop-in replacement. In which case, there's no point in breaking reverse compatibility with every HTTP-capable device on the planet just to make some fussbudgets happy that the URL is "more correct" in some obscure technical sense.
As a counterexample, look at how long we've been working on IPv6. [1] We've been working on that for 20 years, and we're not even close.
And all that aside, URLs aren't primarily API. They're UI. You don't go messing with a UI that billions of people use without a strong reason to do so.
> The only way an upgrade this big will happen is having a single server that speaks both, one that's a drop-in replacement. In which case, there's no point in breaking reverse compatibility with every HTTP-capable device on the planet just to make some fussbudgets happy that the URL is "more correct" in some obscure technical sense.
Why would this preclude using a different protocol in the URI?
> And all that aside, URLs aren't primarily API. They're UI. You don't go messing with a UI that billions of people use without a strong reason to do so.
You mean UI that hidden by most browsers by default now?
Hi! You get one round of point-sniping. After this, make a coherent point if you want a reply.
It wouldn't preclude using a different protocol. It means that there's no point. If every HTTP/2 server is also an HTTP/1.1 server offering the same content then there is no practical need to distinguish the protocols.
As to URL-hiding, you know that's controversial, and for good reason. Their theory is that a lot of people don't normally look, and that could even be true. Many always will, though.
More importantly, that's not the only place that people use links. Every time they share one via email, for example. Every time they type one in from a magazine or the side of a bus or a flyer. Every time they cite one in a paper.
Forcing those people suddenly include a protocol distinction that is meaningless to them and unimportant to the server involved would be a pointless waste. HTTP/2 is purely an under-the-hood improvement. They didn't have to change the URL for SPDY, and they don't have to change it for this.
> If every HTTP/2 server is also an HTTP/1.1 server offering the same content then there is no practical need to distinguish the protocols.
But not ever HTTP/1.1 server will be an HTTP/2 server; that's my point.
> As to URL-hiding, you know that's controversial, and for good reason.
I know; I hat it. I brought it up because obviously most people don't seem to care enough about it that we should overload a schema specifier for the sake of aestetics that don't matter.
> More importantly, that's not the only place that people use links. Every time they share one via email, for example. Every time they type one in from a magazine or the side of a bus or a flyer. Every time they cite one in a paper.
OK? some people cite ftp links. I'm not seeing your point.
> Forcing those people suddenly include a protocol distinction that is meaningless to them and unimportant to the server involved would be a pointless waste.
So we shouldn't use ftp:// or smb:// anymore? I don't get your point, because people are already used to non-http schemas.
> HTTP/2 is purely an under-the-hood improvement.
I mean you could say the same thing if all your assets were served over ftp and the browser rendered them as it does http. HTTP/2 is _fundamentally_ different than HTTP/1.1.
> Or you just run two servers (or a single server that speaks both), you know, like you'll have to for the next 30 years anyway.
The point is not that origin servers need to be upgraded, it's that clients and intermediaries should not be forced to upgrade. That means the actual content of the web (HTML pages) should not require revision to support the wire protocol upgrade, especially considering HTTP has an upgrade mechanism already built in.
There's almost no benefit that I can see to require a new scheme, and tremendous downside.
> That first part, before the colon, is a protocol descriptor. It would be perfectly reasonable to change the protocol descriptor.
And then you look up the spec for how to interpret and/or resolve the rest of the URI. That may or may not involve a mapping to a wire protocol (e.g. the "about" scheme, or the "geo" scheme).
> The point is not that origin servers need to be upgraded, it's that clients and intermediaries should not be forced to upgrade.
And HTTP2 doesn't require these upgrades? Clients and caching servers all need to support the new protocol.
> That means the actual content of the web (HTML pages) should not require revision to support the wire protocol upgrade,
Why would they? I'm not sure why the content matters in this discussion.
> And then you look up the spec for how to interpret and/or resolve the rest of the URI. That may or may not involve a mapping to a wire protocol (e.g. the "about" scheme, or the "geo" scheme).
OK, fair enough. How does this change my point though? HTTP vs HTTP2 requires a change in how the client "interpret[s] and/or resolve the rest of the URI."
The semantics of the URL aren't different - just the protocol version details. You didn't want different schemes for http/1.0 vs http/1.1 right? The URL still represents the same resource and if the server would like to inconsistently use different versions (perhaps because of inconsistent clusters or cdns) that's fine as long as the resource is consistent.
It has the benefit of giving us separate protocols for the transport and application layers. And of already existing, though it isn't widely deployed yet.
But it would allow multiple HTTP requests to be multiplexed over a single connection.
SCTP is also using some form of slow start congestion control, but since it would multiplex the HTTP requests over a single connection there would only be one initial slow start for all the requests.
Why worse? Wouldn't multiple HTTP/1.1 requests sharing a persistent TCP-connection also only have one initial slow start phase?
Maybe the SCTP multiplexing / parallelization of the requests and thus the initial headers affect this negatively as more headers would be transferred during the one initial slow start.
If the delay caused by slow start is a big problem one could add header compression to HTTP/1.x and run it over SCTP.
I understand that a transport protocol designed especially for and embedded in HTTP/2 will be more optimal than a generic one like SCTP. But my argument is that a multiplexing transport protocol like SCTP could be good enough. And usable for more than HTTP/2. And the focus could be on simplifying HTTP instead.
This page appears to be Twitter branded at certain window sizes. Specifically, if the Twitter icon in the header happens to show up more or less centered, it looks just like the header on Twitter (when you're logged in). Maybe use text for that link, or have another logo in the header somewhere?
> Spriting, data-inlining, domain sharding, and concatenation. These hacks are indications of underlying problems in the protocol itself.
While this is true, I don't think HTTP/2 will or should negate some of these. Precomputing things like minification and concatenation will certainly reduce load and always increase capacity. Human-readable code is only useful for humans, and during transfer, storage, parsing, etc., it's just plain inefficient. Also, with spriting and concat, fewer files means more of the information you know you need is at the same place in disk or memory, speeding access at a low level.
Amusingly, many of the design decisions of HTTP/2 were made the same way in Macromedia Flash. Binary representation and multiplexing of streams to improve the user experience are both used in Flash.
Approximately half of Flash's design decisions and APIs were extremely well-designed and well-engineered. There are still many things that Flash does better than pure HTML and their view framework API was actually quite lovely.
That said...the other 50% was frequently so bad that it more than cancelled out the benefit on the other side. Which is why no one uses Flash anymore [1].
I mean multiplexing, why not, flow control is more the job of TCP, that I agree.
But if the goal is network performance and relieving stress on servers and routers and wireless internet phone equipments, I don't think it's such a bad idea. If CPUs are faster and have more cores, it might be good to let CPU do more.
What I was saying is that the current problems this link is talking about, that http would try to solve, those problems are quite gore.
To me the issue with multiplexing is that it's multiplexing "streams", not simply sending multiple files.
(There is complete support for MIME multipart in browsers already, but if that were fixed it could be useful for "pushing" content from the server).
> What I was saying is that the current problems this link is talking about, that http would try to solve, those problems are quite gore.
I just don't think that HTTP2 (that name still kills me as this protocol is as much HTTP as clean coal is clean) really makes for a cleaner protocol. I think that it's being sold as HTTP is what really bothers me to the core -- it's not a replacement for HTTP; it's a Layer 4 protocol implemented at Layer 7. That's fine, I guess, but I don't think that it's inherently "simpler" than a Request-Response, text-based protocol (which, admittedly has been used to kludge a lot of things).
Also, I was hoping HTTP would have solved actual issues, and not just those faced by heavy-weight websites. Issues such as:
* Better authentication
* More secure caching
* Better methods to find alternate downloads locations
* Keeping the protocol simple
* Making each request contain less information about the sender
* Improved Metadata
71 comments
[ 2.2 ms ] story [ 118 ms ] thread1. The well-known resilience of mail can be largely attributed to its use of MX records to specify the use of backup servers and/or load balancing. SRV records are essentially a generalization of MX records for any service, not just mail, so using SRV records would bring this benefit to any protocol which uses them.
2. The problem of “www” or no “www” would be entirely eliminated; no need for non-standard so-called ANAME or ALIAS “records”. Just imagine – if, in an alternate universe, mail servers hadn’t used MX records, it would then have been necessary to have an A record on the bare domain name, and later, when the Web was created in this alternate universe, the web server would have to be the same server as the mail server, since both now would use the bare domain name. In our universe, this position is now solely occupied by the HTTP protocol, which causes lots of pain in the DNS when redundant (and harder to update) A records need to be put on the bare domain name just for HTTP. Not to mention IPv6, which, for HTTP to work, requires an additional AAAA record on the bare domain.
3. The “everything is on port 80, let’s reinvent everything on top of HTTP” problem could certainly have been entirely avoided, and could be slowly fixed, by using SRV records. (A server concerned about clients needing to traverse old-style restrictive firewalls could provide a low-priority port 80 server as a fallback.)
① A DNS RR for specifying the location of services (DNS SRV) https://tools.ietf.org/html/rfc2782
I agree it's a pretty decent system, and something I'd like to see more systems adopt it.
Interestingly, I can't actually think of a reason why a (customised) web browser couldn't use it now. If a browser understood SRV records it could use the correct port. There might be some odd interactions with the HTTP 1.1 Host header, but I haven't thought about it much.
Of course, browsers that don't understand it wouldn't be able to follow links that use it which is something of a downside I guess.
(I wrote about this previously here on HN: https://news.ycombinator.com/item?id=7522208)
Indeed, since the HTTP/1.1 RFC was issued in 1999 and the SRV RFC in 2000 there is no way it could have mentioned it.
If there are technical reasons why it can't work that's one thing. If some verbiage in the RFC is all that holds it back it's easy to fix: try it out, then issue a new RFC.
The whole RFC process embraces changes like this.
And an RFC is not a standard, and all life is meaningless? No, a SHOULD NOT should be taken seriously, not ignored.
The SRV RFC included that language for a reason, and probably had HTTP in mind when they wrote it, and therefore wanted HTTP clients not to use SRV records randomly. Since then, HTTP has has plenty of time to add SRV usage in its minor revisions, but has chosen not to. But now, with HTTP/2, a major revision, it is the perfect time to add this kind of thing.
RFCs are not like ISO standards. The name says it all: "request for comments".
RFCs reflect the ideal of "rough consensus and running code".
The SRV RFC was written by an author from Microsoft aimed at domain controller discovery. They probably didn't think of HTTP at all.
There's an expired draft RFC for SRV+HTTP[1].
You are right: SHOULD NOT should be taken seriously. But if you are (say) Mozilla, Microsoft or Google and writing a webbrowser you probably know enough to understand when you can break a RFC.
How do you think TCP-fast-start happened?
[1] http://tools.ietf.org/html/draft-andrews-http-srv-01
The SRV RFC was not by Microsoft. It has three authors, the other two being one Arnt Gulbrandsen from Troll Tech (of the Qt library fame) and Paul Vixie of the Internet Software Consortium, who is also the author of BIND. I actually am inclined to think that they did consider HTTP seriously.
This was discussed by the WG. The best rationale is given in the thread starting from this mail[0] from Will Chan at Google, followed up by this one[1] from Amos Jeffries (Squid Proxy). The summary is: waiting for SRV lookups can be slower than just doing straight-up A records, so no client in a competitive space is going to do it.
Everyone agrees that in an ideal world this would be great, but in practice no-one is going to be the first person to move on that front. The wins aren't big enough and the cost is too high.
[0]: http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/... [1]: http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/...
Reading that I'm not sure the problem is that SRV lookups are slower, but that forcing clients to do a SRV lookup, wait, and then a A/AAAA queries is slower:
If I own a popular client and I switch to mandatory SRV lookup, and don't issue A and AAAA queries until the SRV lookup fails, then for the services and paths where SRV lookup doesn't work, user experience will be vastly degraded.[1]
However,it appears that making SRV optional remains a possibility:
Note that none of what I've said necessarily applies to optionally doing SRV and using it when the results are available.
[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/...
Few if any resolvers support this feature, but it show that the problem we have is because of legacy assumption that are no longer true. Fix those, and the problem goes away.
: https://www.ietf.org/rfc/rfc1035.txt
If the language says "You MUST consult SRV records for _http2._tcp before you use the result of an A lookup", then anything that delays that SRV lookup will delay page resolution. Browsers are strongly disincentivised from making anything slow, so it's just safer not to add the extra delay point. If we specced that, no-one would do it.
Better to have what we have now, which is: do SRV if you want.
What Amos was talking about is the problem of some resolvers not returning an answer at all, not even NXDOMAIN or an empty authoritative answer, due to the fact that he queried for AAAA records and some resolvers are buggy and can’t handle those queries. SRV is irrelevant to this problem, except that using “happy-eyeballs” and requiring A and AAAA has problems.
What William Chan (陈智昌) was talking about was where a query for SRV records fail, causing a long timeout. But this is not the normal case; normally you get an answer (perhaps an empty one) immediately. And since this has nothing to do with IPv6 and buggy resolvers, I don’t know why this situation should be considered common enough to optimize for. (Clients which really wanted to mitigate this could certainly detect this case when it happens and keep a blacklist.)
You keep implying that such things will happen. They won’t. There is no reason for an SRV record lookup to delay page resolution any more than an A (or AAAA) record lookup.
> If we specced that, no-one would do it.
Are you seriously saying that even if the HTTP/2 standard came out and it said “A client MUST lookup an SRV record and use it if one exists, and not use any A or AAAA records before the non-existence of an SRV record can be confirmed.”, that the browser vendors would simply ignore this? If that’s the case, why do we even have a standards process? Why do we even pretend to write RFCs if we can just assume that every implementer will ignore the standard and do whatever they feel like?
No, a thing which would be a net win for the Internet in general should go in the standard. RFC writers should not bow to fears of what Google, IBM, or General Motors might or might not do. Or, at least, wait until they make the threat of non-compliance openly.
I regard this as the worst-case scenario. It is a baffling omission. The HTTP URI scheme [RFC7230] does nothing to clarify matters:
"If host is a registered name, the registered name is an indirect identifier for use with a name resolution service, such as DNS..."
In my view the WG has the opportunity to definitively fix the problems of dual-stack discovery and of apex record service names, and the refusal to do so is exceedingly disappointing. We're left with implementations of Happy Eyeballs that have lookup race conditions and therefore nondeterministic behaviour.
All the arguments against using service records are actually complaints about artefacts arising from the current ambiguity in resolution practices, and thereby perpetuate the problem.
The proper choice should be that all IPv6 sites should be resolved via SRV rather than raw AAAA lookup - this is the soft introduction that over the years will ramp up to a global first preference for service records, and remains compatible as a variation on Happy Eyeballs.
But entrenched positions don't change and I have no clout with the WG or with browser product managers, so I gave up.
SPDY and HTTP/2 can multiplex their data streams across a single connection. That is to say, they can receive 2+ datastreams on the same TCP connection at the same time.
Their more involved properties require dozens to even hundreds of requests in order to perform complex tasks like showing a list of plain text posts.
And, of course, they can't possibly impose any kind of discipline on the way they do things, so instead the entire internet has to change to accommodate them.
Now I could optimize that even more by inlining everything with data-inline: but that would impact caching code and be strictly worse for the network.
A property from google even websearch has fancy transitions and animations and dynamic content. And Google has every incentive to limit the number of connections to their page. When I worked there I worked on some of those technologies specifically to avoid too many connections.
Blaming Google for not being disciplined enough disregards their economic incentives to be disciplined enough. They invested a lot of energy in this before SPDY came out. There comes a time where you just have to recognize that the protocol is fighting against you and either give up or evolve the protocol.
(1) index.html, (2) app.nocache.js (3) A56284CFC0D680C7DBA0F93CA9AE8F75.cache.js (cache fragment for given page/state) (4) XHR for page data/state to display
Now, if you've got some CSS and minor images/fonts, these can be an your options request (5) resource bundle. On many Google properties, (2 and 5) are inlined into (1).
The take away? These technologies and frameworks have been out there for a long time (8 years in the case of GWT), so you don't have to make hundreds of requests if you're disciplined. I can only assume Google has thrown in the towel trying to get script kiddies to ever be disciplined.
No, wait a sec. That doesn't sound right. Could it be that our tools exist to serve humans? And that rather than putting effort into accommodating ourselves to the limitations of our current tools, we should instead put our effort into making tools that work the way we want?
Nah, sounds crazy. If HTTP/1 was good enough for my grand-dad, it's good enough for me.
What about tools for reading corrupt binary HTTP/2 frames?
The rare occasion where this will be a big difference for me is telnetting to port 80 and typing HTTP by hand. Honestly, I only do that out of habit; I can do the same things just as well with curl. But even when HTTP/2 ships, I'm sure it will be at least a decade before servers stop supporting HTTP/1.1. Probably longer. So I'm not expecting this to make a big difference to my debugging habits.
Have they actually worked out how to implement a HTTP/2 front end cache in the face of ill-behaved clients, crashy servers, and the way wireless breaks a lot of the assumptions that would make "one big TCP session" a good idea in the first place?
If TCP+TLS handshakes are so expensive, wouldn't it be a better investment of everyone's resources to work on an L3 protocol for setting up a fast second session? Leave the multiplexing to packet people.
A client can say it does not want push in the settings frame up front. And then it can make one request at a time, effectively avoiding multiplexing, although each frame in the response will have a 9-byte frame header (similar to the overhead in chunking today in 1.x).
By saying it does not want push, it relieves itself of having an object cache. It will still be on the hook for having an HPACK table as state. (In practice this will be coded once in each language in a library and most developers will never think about it.)
> Have they actually worked out how to implement a HTTP/2 front end cache in the face of ill-behaved clients, crashy servers, and the way wireless breaks a lot of the assumptions that would make "one big TCP session" a good idea in the first place?
The community has indeed been working on a corpus of interoperability libraries, servers, clients, and proxies: https://github.com/http2/http2-spec/wiki/Implementations
> If TCP+TLS handshakes are so expensive, wouldn't it be a better investment of everyone's resources to work on an L3 protocol for setting up a fast second session? Leave the multiplexing to packet people.
It's multiplexing with declared weighting and dependency trees. So for example, the client can tell the server that it wants files A, B, C, and D, but to have B, C, D is useless until it has A, and once A is finished, the relative importance of B is twice as much as C or D. That's much easier to do on one socket.
Also one socket gets the header compression in HPACK to benefit from all the headers going through the same state engine. Giant sets of cookies (more common than we'd like to believe) are now more efficiently expressed.
One socket also helps the server-side keep statistics and logs more organized as related requests are together.
I understand your suggestion to do it on a lower level, and that's a fine idea, but I also understand why the working group did it on an HTTP layer, especially with SPDY as precedent.
I work at Akamai, and I'm one of the open source library contributors (the Common Lisp one). These opinions are my own.
Granted, I think it was overall a poor decision to introduce that scheme prefix as well. It's just that, many arguments against a new scheme prefix also applied in that case, and many arguments for that new scheme prefix would apply in this case.
Further, technically speaking, the URL scheme has little to with the protocol used , it is merely a way of linking a hierarchical namespace to a spec (or more) that tells a developer how to resolve it. That means we have a lot of flexibility in terms of protocol (we don't even need a protocol defined in some cases - mailto: URIs for example).
Or you just run two servers (or a single server that speaks both), you know, like you'll have to for the next 30 years anyway.
> and would create a separate web with content that existing clients or intermediaries would be excluded from.
If you have an HTTP2-only site, aren't you doing that anyway?
> Much more evolvable to keep the wire protocol version orthogonal to URI.
> Further, the URL scheme has little to with the protocol used
That first part, before the colon, is a protocol descriptor. It would be perfectly reasonable to change the protocol descriptor.
What should have been done was use the Upgrade header (like websockets) or specify using SRV records.
As a counterexample, look at how long we've been working on IPv6. [1] We've been working on that for 20 years, and we're not even close.
And all that aside, URLs aren't primarily API. They're UI. You don't go messing with a UI that billions of people use without a strong reason to do so.
[1] http://www.ipv6.com/articles/general/timeline-of-ipv6.htm
Why would this preclude using a different protocol in the URI?
> And all that aside, URLs aren't primarily API. They're UI. You don't go messing with a UI that billions of people use without a strong reason to do so.
You mean UI that hidden by most browsers by default now?
It wouldn't preclude using a different protocol. It means that there's no point. If every HTTP/2 server is also an HTTP/1.1 server offering the same content then there is no practical need to distinguish the protocols.
As to URL-hiding, you know that's controversial, and for good reason. Their theory is that a lot of people don't normally look, and that could even be true. Many always will, though.
More importantly, that's not the only place that people use links. Every time they share one via email, for example. Every time they type one in from a magazine or the side of a bus or a flyer. Every time they cite one in a paper.
Forcing those people suddenly include a protocol distinction that is meaningless to them and unimportant to the server involved would be a pointless waste. HTTP/2 is purely an under-the-hood improvement. They didn't have to change the URL for SPDY, and they don't have to change it for this.
But not ever HTTP/1.1 server will be an HTTP/2 server; that's my point.
> As to URL-hiding, you know that's controversial, and for good reason.
I know; I hat it. I brought it up because obviously most people don't seem to care enough about it that we should overload a schema specifier for the sake of aestetics that don't matter.
> More importantly, that's not the only place that people use links. Every time they share one via email, for example. Every time they type one in from a magazine or the side of a bus or a flyer. Every time they cite one in a paper.
OK? some people cite ftp links. I'm not seeing your point.
> Forcing those people suddenly include a protocol distinction that is meaningless to them and unimportant to the server involved would be a pointless waste.
So we shouldn't use ftp:// or smb:// anymore? I don't get your point, because people are already used to non-http schemas.
> HTTP/2 is purely an under-the-hood improvement.
I mean you could say the same thing if all your assets were served over ftp and the browser rendered them as it does http. HTTP/2 is _fundamentally_ different than HTTP/1.1.
The point is not that origin servers need to be upgraded, it's that clients and intermediaries should not be forced to upgrade. That means the actual content of the web (HTML pages) should not require revision to support the wire protocol upgrade, especially considering HTTP has an upgrade mechanism already built in.
There's almost no benefit that I can see to require a new scheme, and tremendous downside.
> That first part, before the colon, is a protocol descriptor. It would be perfectly reasonable to change the protocol descriptor.
Actually, if you look at the specs, the first part is a scheme identifier, not a protocol identifier. There's a big difference. Schemes point you here: http://www.iana.org/assignments/uri-schemes/uri-schemes.xhtm...
And then you look up the spec for how to interpret and/or resolve the rest of the URI. That may or may not involve a mapping to a wire protocol (e.g. the "about" scheme, or the "geo" scheme).
And HTTP2 doesn't require these upgrades? Clients and caching servers all need to support the new protocol.
> That means the actual content of the web (HTML pages) should not require revision to support the wire protocol upgrade,
Why would they? I'm not sure why the content matters in this discussion.
> And then you look up the spec for how to interpret and/or resolve the rest of the URI. That may or may not involve a mapping to a wire protocol (e.g. the "about" scheme, or the "geo" scheme).
OK, fair enough. How does this change my point though? HTTP vs HTTP2 requires a change in how the client "interpret[s] and/or resolve the rest of the URI."
I think the better route is to investigate how HTTP/1.1 could be layered atop multiplexing transport protocols like SCTP.
It might take 10 years to reach wide deployment, but HTTP/1.1 over TCP is working just fine anyways.
And as the FAQ says, if you don't use header compression then response headers alone will take many RTTs to transmit due to slow start.
But it would allow multiple HTTP requests to be multiplexed over a single connection.
SCTP is also using some form of slow start congestion control, but since it would multiplex the HTTP requests over a single connection there would only be one initial slow start for all the requests.
Which is worse than HTTP/1.1. Avoiding that problem is why HTTP/2 uses hpack.
Maybe the SCTP multiplexing / parallelization of the requests and thus the initial headers affect this negatively as more headers would be transferred during the one initial slow start.
If the delay caused by slow start is a big problem one could add header compression to HTTP/1.x and run it over SCTP.
I understand that a transport protocol designed especially for and embedded in HTTP/2 will be more optimal than a generic one like SCTP. But my argument is that a multiplexing transport protocol like SCTP could be good enough. And usable for more than HTTP/2. And the focus could be on simplifying HTTP instead.
While this is true, I don't think HTTP/2 will or should negate some of these. Precomputing things like minification and concatenation will certainly reduce load and always increase capacity. Human-readable code is only useful for humans, and during transfer, storage, parsing, etc., it's just plain inefficient. Also, with spriting and concat, fewer files means more of the information you know you need is at the same place in disk or memory, speeding access at a low level.
That said...the other 50% was frequently so bad that it more than cancelled out the benefit on the other side. Which is why no one uses Flash anymore [1].
[1] Everyone still uses Flash.
I mean multiplexing, why not, flow control is more the job of TCP, that I agree.
But if the goal is network performance and relieving stress on servers and routers and wireless internet phone equipments, I don't think it's such a bad idea. If CPUs are faster and have more cores, it might be good to let CPU do more.
What I was saying is that the current problems this link is talking about, that http would try to solve, those problems are quite gore.
(There is complete support for MIME multipart in browsers already, but if that were fixed it could be useful for "pushing" content from the server).
> What I was saying is that the current problems this link is talking about, that http would try to solve, those problems are quite gore.
I just don't think that HTTP2 (that name still kills me as this protocol is as much HTTP as clean coal is clean) really makes for a cleaner protocol. I think that it's being sold as HTTP is what really bothers me to the core -- it's not a replacement for HTTP; it's a Layer 4 protocol implemented at Layer 7. That's fine, I guess, but I don't think that it's inherently "simpler" than a Request-Response, text-based protocol (which, admittedly has been used to kludge a lot of things).
Also, I was hoping HTTP would have solved actual issues, and not just those faced by heavy-weight websites. Issues such as:
(I get into those a little more here: https://github.com/jimktrains/http_ng)