In their defense that is probably the case of their MVC framework or CMS returning a 200 after not being able to match the route and not really them doing it on purpose.
But yeah, I agree- (though I've missed them I'm sure) I pretty voraciously make sure my HTTP status matches my intended response- and sometimes that means you've got to write the code in the controller that tells the request to specifically return a certain status.
That just pushes "web professionals not understanding HTTP" one level down the stack, except this "web professional" is not the user doing the clicky, but whoever made the framework; the problem remains unchanged.
Sometimes it's a disagreement over what it's reasonable to expect from a framework user. Framework developers expect the users to read their docs and use their API only the way that's documented. Users expect any given API method to do what it says.
Heck, it might even be more common. Especially when you talk about huge frameworks like Spring for Java. Also, with CMSes like Wordpress I imagine you can't really change the status that easily if it's not returning what you'd expect. I'm not saying a WP 404 page DOESN'T return 404, but if it returned 200, how far would you have to dig to change that? If my CMS wasn't doing it exactly right, I would probably just shrug and move on unless it was critically important to something I was writing against it.
Request (section 5) and Response (section 6) messages use the generic
message format of RFC 822 [9] for transferring entities (the payload
of the message). Both types of message consist of a start-line, zero
or more header fields (also known as "headers"), an empty line (i.e.,
a line with nothing preceding the CRLF) indicating the end of the
header fields, and possibly a message-body.
/quote RFC2616
(So, the status line is an entity of its own, which is followed by headers.)
At least on three counts - in user agents, in crawlers and in RFC:
- I definitely see bad design in interpreting the RFC as "if no caching metadata, then cache forever;" this violates Principle of Least Astonishment.
- Also, "301 and 302 has completely different meaning to a crawler" seems illogical.
- Perhaps the RFC should have also specified something like "in absence of caching headers, don't assume 'cache forever', that is a long time"; it does not try too much to prevent implementors from shooting themselves (and users) in the feet.
See my original post - permanent Permanent Redirect works as designed ;) I was later trying to point out possible ambiguities in how the spec is interpreted vs. how it's written.
In defense of the browsers - the RFC does say, in essence, "rewrite the 301'd URL and be done with it," so the browser does not even need to deal with the redirect any more.
I would argue that your statement is only true as long as the URL in question is owned by the application that sent the 301 in the first place. If a different application now owns the URL, then it violates least astonishment for the URL to be redirect when, from it's perspective, the redirect was never sent.
At the least, a redirect should not be permanent across name server changes (when the domain changes hands). Unfortunately, this would effectively be adding state to HTTP.
> If a hacker (from a competitor) 301 redirects your domain, you're toast.
Reminds me of the guy that "got hold" of Google's domain for a few minutes. What if something similar happened and someone were able to make this redirect? Millions would be affected.
Nope, IIRC he didn't technically own the domain, it just showed up in his account as owning it (therefore he could have viewed analytics and other information) - but he didn't actually have permission to change the domain.
Not only that, anybody with a WiFi Pineapple or any other MITM could 301 redirect anything you visit without HTTPS to their spam site.
It seems like browsers interpreting permanent as forever is some kind of a bug. Even if that's literally what it says, that's not what anybody wants. What great evil is being prevented by not having it expire and be refreshed after 45 days?
As a developer, I would definitely consider it unintended behavior if a browser interprets "Page Moved Permanently" to mean "URL Permanently Discontinued".
That fixes the problem for you as a web client, but not you as a domain owner. Everyone who visited it during the 301 will continue to treat it as such until they clear their cache (or equivalent).
It's not indefinite because you need to specify a duration. However nothing stops you from setting an extremely long duration and in fact most tutorials seem to advise doing so for safety reasons.
"There are only two hard things in Computer Science..."
I guess some people think the purpose of 301 is more like that of 410: update references so you don't try to go there again. The difference is that with 301 you additionally instruct the client to not even attempt to go there again in the future.
But the article does raise an interesting point: if I own somedomain.example and set it up with a 301 redirect to myotherdomain.example and enough people visit it that most people will have cached the redirect, doesn't that basically mean I now own it for perpetuity (or until enough people have cleared their cache) even if I don't renew the domain and new requests to the domain are no longer served (by the same IP)?
Or do browsers have some kind of protections against this, at least based on DNS? It's a bit too convoluted for a proper DOS attack (because you need to own the domain long enough and make it popular enough to poison everyone's caches) but a naive implementation seems like it would effectively render domains unusable if someone set up a 301 on them at some point in the past.
For an important busy web site, even serving up a redirect for a short amount of time could be enough to cause some serious problems. It's a way of turning "I hacked and defaced this website but they fixed it 24 hours later" into "I hacked and defaced this website and they fixed it 24 hours later, but loads of people still see the defaced version".
I took over a domain which had previously 301-redirected HTTP:// to HTTPS://. It caused us no end of trouble getting the alpha site online -- obviously we set up SSL but we didn't realise it was the _first thing we'd have to do_.
It also caused half a day of confusion to understand why some of our web browsers were still failing to connect and others could see the alpha site (because they'd never visited the previous 301 site at that address).
This isn't just a problem with things like HTTP. The industry as a whole lacks a standard uniform way of dealing with domain transfers or expiration. CAs for example will happily issue certificates that expire after your domain.
I mean, would it not be a problem for the CA that you (after the domain has expired) still is in possession of a valid certificate for a domain which you do not have control over?
> It's not the CA's job to make sure your domain isn't expiring. It's yours.
Really? In the simplest case, their entire job is certifying that the holder of the private key is the holder of the domain name[1]. That begs the question, of course: how is it that we trust every single CA to certify every single domain? Why don't we trust the issuer of each domain hierarchy to certify only those domains it's permitted to issue?
The entire XPKI is broken, broken, _broken_.
[1] In the more complex case, of course, they certify that the keyholder is some external entity.
This was probably HSTS not a 301. HSTS headers include a validity, but 1yr is common. Good luck convince anyone to undo this, clearing cache does help and you need to dive into your browser internals for a fix: chrome://net-internals/#hsts
I was recently burned by this. The client had left his server misconfigured for a few hours, and a lot of his static content ended up in a redirect loop. I was called in to fix the mess.
Although modern browsers are clever enough to detect a redirect loop and throw an error, they're not clever enough to detect when the redirect loop is caused by a cached 301 response. So they cache the redirect loop as well. Throw in another layer of caching (CloudFlare), and now you've got a bunch of URLs that will be stuck in a redirect loop for a very long time.
The only solution was to append some garbage to every URL, like "?cache=no". Fortunately, the problem only occurred with static content, so nginx happily discarded the querystring and returned fresh content.
2012 - somebody should write 2012 into the title of this post (that by the way hasn't any concrete data)
I did some testing in 2009, think around 2012 and 2014. Additional to loffilegrepping after some big site URL rewrites.
It's a non issue. No caching headers, the redirect gets cached only for the current browser session. Close it, reopen it, gone, done.
Lets discuss this one based on data. (Which I cant provide right now as Im on a beach on Sri Lanka right now with a FirefoxOS device amd I dont know how to see Http requests on this one, but) Please prove me wrong! based on test, data, not blogposts.
I just tested it in Firefox on a Mac. I restarted Firefox. I even rebooted. Developer Tools > Network tab says "cached". I can't confirm that it is cached forever, but it is not only "for the current browser session".
Here's the relevant Gecko (Firefox) code which tries to use the max-age and expires headers first and then will set it to forever if the response code was 300, 410, 301, or 308. Note that I'm going by a somewhat shallow code reading after a recent investigation. There's a lot of stuff going on in gecko/necko that could potentially apply some failsafe time limit, maybe in the cache implementation, so I wouldn't take this as 100% for sure. Breaking on this function with gdb and tracing the flow is probably a better idea if you really want to know.
Why not just redirect back from the new to the old URL? A sane client should then check if the old 301 is still there (not sure if by the spec, but it is common sense). I believe I once tried this and it worked with Firefox. Not sure, though.
When working with non-trivial redirects, test them with something like "wget -S $url" instead of your browser. The redirect caching makes it very painful to to repeatedly test the same redirect.
Also, start with a 302, and only change the status to 301 once you're confident they are correct.
This can be fixed with a 302 from the destination back to to the start/source of the 301, assuming that the old domain isn't redirecting any more (which would cause a redirect loop)
It's easy to live your life to keep Google happy. 301 redirects have been an important part of online website life.
But then I do accept this perspective (if it's within your call to take this risk). Just don't 301-redirect it. Let the search engines figure it out for themselves.
If a user has a bookmark to an old resource, then it's a liability for you to try to keep your web of 301s working.
I would say links are a bigger problem than bookmarks. For instance stack overflow answer links to apple or msdn references, are really annoying if they die.
Likely a site with the influence of twitter would be able to get this taken care of. (directions at another domain on how to clear your cache or even a direct update from browser vendors possibly)
This is a browser design issue not a protocol/server issue. At the time HTTP 1.1 was written browser caches would hold at most a few weeks of browsing history before things started falling out so 'permanent' functioned like our intuition
Whats more surprising is "13.2.2 Heuristic Expiration"[1]
If you specify (or your framework does) a Last-Modified time WITHOUT Cache-Control the browser is free to make up its OWN cache expiration rules (the item is implicitly cacheable)
Heuristic Expiration is a case of documenting existing weirdness. Older browsers would have cache expiration policy be a setting the user could control, so you can't really leave off explicit headers with any expectation a default will be respected.
To save you a click: Jeb Bush forgot to renew his domain,
and Trump bought it and redirected to
www.donaldjtrump.com
Thankfully, Trump is using a 302, not a 301.
--- end factually incorrect statement ---
EDIT: ah I see -- jebbush.com has only ever had that redirect throughout its entire (short) existence. Leaving my comment up though, since it's more about what one could do in that scenario than any particular current event.
I thought best practice these days, at least for REST, was a 308 (https://tools.ietf.org/html/rfc7238), since 301 has the bizarre behavior of being converted to a GET by some UAs (which conflate it with 303), enshrined for hysterical raisins. Is this not the case?
If the owner of the domain had done it, would it still really be an "eternal issue" since:
a. most popular websites nowadays are so bloated that browser caches would throw out many things (including your tiny site that someone checks a few times a week or even longer) a lot sooner compared to how things were about 10 years ago (I presume the default disk cache sizes in browsers have not increased by multiples in this period).
b. more people are browsing through mobile devices that are dumped in a few years and replaced with a new one, new browser, empty cache, etc.
Is there any company that provides 301 or 302 as a service? -- something cheap. I know it's not hard to set-up a small box on AWS and install Node.js (or whatever). But I would pay few dollars a month for some service to run that for me.
91 comments
[ 3.1 ms ] story [ 150 ms ] threadAlso, "This response is cacheable unless indicated otherwise," says RFC 2616.
Working as designed, IMNSHO. Perhaps not working as intended, but alas, that's a case of ¬RTFM.
Much like webpages that say "404 not found" with a "200 OK" header.
But yeah, I agree- (though I've missed them I'm sure) I pretty voraciously make sure my HTTP status matches my intended response- and sometimes that means you've got to write the code in the controller that tells the request to specifically return a certain status.
/quote RFC2616 (So, the status line is an entity of its own, which is followed by headers.)
- I definitely see bad design in interpreting the RFC as "if no caching metadata, then cache forever;" this violates Principle of Least Astonishment.
- Also, "301 and 302 has completely different meaning to a crawler" seems illogical.
- Perhaps the RFC should have also specified something like "in absence of caching headers, don't assume 'cache forever', that is a long time"; it does not try too much to prevent implementors from shooting themselves (and users) in the feet.
In defense of the browsers - the RFC does say, in essence, "rewrite the 301'd URL and be done with it," so the browser does not even need to deal with the redirect any more.
At the least, a redirect should not be permanent across name server changes (when the domain changes hands). Unfortunately, this would effectively be adding state to HTTP.
You often can't use 302 because all your external links no longer work SEO magic for you with a 302. Google only transfers link juice with 301 [1].
If you make a mistake and misconfigure your server, you're toast.
If a disgruntled employee 301 redirects your domain, you're toast.
If a service provider misconfigures your domain, you're toast.
If a hacker (from a competitor) 301 redirects your domain, you're toast.
If you buy a domain that had a 301 on it, it's worthless.
If you buy a domain that had 301s on it that point to phishing sites, you're in trouble.
I always add cache headers to 301 redirects I use to at least prevent me from shooting myself with an arrow in my knee.
UPDATE: [1] Google seems to have changed this recently. It also no longer considers http/https different pages as it did in the past with the same content https://www.searchenginejournal.com/google-confirms-no-loss-...
how any of you hold jobs is beyond me. i'll keep coming in and cleaning up your messes.
http://mark.koli.ch/set-cache-control-and-expires-headers-on...
Reminds me of the guy that "got hold" of Google's domain for a few minutes. What if something similar happened and someone were able to make this redirect? Millions would be affected.
It seems like browsers interpreting permanent as forever is some kind of a bug. Even if that's literally what it says, that's not what anybody wants. What great evil is being prevented by not having it expire and be refreshed after 45 days?
I guess some people think the purpose of 301 is more like that of 410: update references so you don't try to go there again. The difference is that with 301 you additionally instruct the client to not even attempt to go there again in the future.
But the article does raise an interesting point: if I own somedomain.example and set it up with a 301 redirect to myotherdomain.example and enough people visit it that most people will have cached the redirect, doesn't that basically mean I now own it for perpetuity (or until enough people have cleared their cache) even if I don't renew the domain and new requests to the domain are no longer served (by the same IP)?
Or do browsers have some kind of protections against this, at least based on DNS? It's a bit too convoluted for a proper DOS attack (because you need to own the domain long enough and make it popular enough to poison everyone's caches) but a naive implementation seems like it would effectively render domains unusable if someone set up a 301 on them at some point in the past.
It also caused half a day of confusion to understand why some of our web browsers were still failing to connect and others could see the alpha site (because they'd never visited the previous 301 site at that address).
Really? In the simplest case, their entire job is certifying that the holder of the private key is the holder of the domain name[1]. That begs the question, of course: how is it that we trust every single CA to certify every single domain? Why don't we trust the issuer of each domain hierarchy to certify only those domains it's permitted to issue?
The entire XPKI is broken, broken, _broken_.
[1] In the more complex case, of course, they certify that the keyholder is some external entity.
https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
It would have been useful to include those headers in the blog post.
Last-Modified: Fri, 19 Feb 2016 12:54:49 +0100
Expires: Sun, 29 May 2016 12:54:49 +0200
Cache-Control: max-age=8640000, must-revalidate
See also: https://www.mnot.net/cache_docs/#CONTROL
Although modern browsers are clever enough to detect a redirect loop and throw an error, they're not clever enough to detect when the redirect loop is caused by a cached 301 response. So they cache the redirect loop as well. Throw in another layer of caching (CloudFlare), and now you've got a bunch of URLs that will be stuck in a redirect loop for a very long time.
The only solution was to append some garbage to every URL, like "?cache=no". Fortunately, the problem only occurred with static content, so nginx happily discarded the querystring and returned fresh content.
I did some testing in 2009, think around 2012 and 2014. Additional to loffilegrepping after some big site URL rewrites.
It's a non issue. No caching headers, the redirect gets cached only for the current browser session. Close it, reopen it, gone, done.
Lets discuss this one based on data. (Which I cant provide right now as Im on a beach on Sri Lanka right now with a FirefoxOS device amd I dont know how to see Http requests on this one, but) Please prove me wrong! based on test, data, not blogposts.
I just tested it in Firefox on a Mac. I restarted Firefox. I even rebooted. Developer Tools > Network tab says "cached". I can't confirm that it is cached forever, but it is not only "for the current browser session".
https://dxr.mozilla.org/mozilla-central/source/netwerk/proto...
The 301, 308 stuff comes from IsPermanentRedirect which is here: https://dxr.mozilla.org/mozilla-central/source/netwerk/proto...
edit: > A sane client should then check if the old 301 is still there Check out kijin's comment about this.
Also, 301 might poison proxy-caches, so even if you clear the cache in your browser it might still not work.
It would be good to get an indication of the potential impact.
Also, start with a 302, and only change the status to 301 once you're confident they are correct.
But then I do accept this perspective (if it's within your call to take this risk). Just don't 301-redirect it. Let the search engines figure it out for themselves.
If a user has a bookmark to an old resource, then it's a liability for you to try to keep your web of 301s working.
KISS!
Whats more surprising is "13.2.2 Heuristic Expiration"[1]
If you specify (or your framework does) a Last-Modified time WITHOUT Cache-Control the browser is free to make up its OWN cache expiration rules (the item is implicitly cacheable)
[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
--- begin factually incorrect statement ---
--- end factually incorrect statement ---EDIT: ah I see -- jebbush.com has only ever had that redirect throughout its entire (short) existence. Leaving my comment up though, since it's more about what one could do in that scenario than any particular current event.
a. most popular websites nowadays are so bloated that browser caches would throw out many things (including your tiny site that someone checks a few times a week or even longer) a lot sooner compared to how things were about 10 years ago (I presume the default disk cache sizes in browsers have not increased by multiples in this period).
b. more people are browsing through mobile devices that are dumped in a few years and replaced with a new one, new browser, empty cache, etc.