HN Explorer

The 404 Test: Wildly Brilliant or User Suicide? (blog.dougpetkanics.com)

67 points by thedob ↗ HN
Alternate link - http://dob.posterous.com/the-404-test-wildly-brilliant

Apparently the DNS may not have propagated fully. But it's quite ironic that the link is leading to a 404 for some people.

47 comments

[ 2.5 ms ] story [ 109 ms ] thread
jzell
I really like this philosophy. Why waste precious development time early on building something no one wants?
andreyf
The philosophy is a truism, but the methodology is flawed - users don't know if they need a feature until they've used it.
ken
"A website with intentional 404's sprinkled around" is also something no one wants.
45454564654
Unless it turns out that people really do want it. In which case you've just pissed off hundreds or thousands of your potential early adopters with a 404 page, and demonstrated that you can't even build and link to a simple, basic, web page. And, come on, who here does not hit F5 when they reach a 404 page? I've never heard of anything so stupid in my life - the guy who dreamed this up has just got to have an MBA.
dandelany
A solid idea, but why not spend 5 more minutes, make the link point to a page that says "Sorry, we're still working on [feature X]. Check back soon!" and avoid looking like an amateur?

Better yet, include a form on that landing page asking how important the feature would be to the user, and get the best of both worlds.

trickjarrett
I completely agree. It's much better to test for interest in the feature in the works with a page saying such.
DLWormwood
> make the link point to a page that says "Sorry, we're still working on [feature X]. Check back soon!" and avoid looking like an amateur?

You've officially missed the point of the article. The argument is not that you look like an amateur, but that the user thinks the problem is with their computer or ISP. Since most 404's are due to user error in URI entry, web filters or cross-site link rot, this is a reasonable assumption on the part of the user, an assumption that can be readily exploited.

dandelany
I got that, I just don't necessarily agree with it. IMHO, if you've got a web app that users are savvy enough to know that they need, especially one that costs money, 9 times out of 10 they're going to be savvy enough to know that getting a 404 page with your branding on it is your company's problem, not theirs.

I know this because I work for a company that provides an enterprise B2B web app, and customers already consistently blame the stuff that ISN'T our problem (connection, filters, ancient browser, javascript turned off, etc.) on us.

And yes, a lot of 404's are due to improper URI entry and "cross-site link rot"... but isn't that a completely moot point since we're talking about internal links?

breck
Okay. I've only seen this done with major consumer websites with high traffic. For B2B sites, I think you are right. Your solution would probably work better.
nostrademons
Maybe I've just got more tech-savvy friends than most, but I don't know anyone who assumes a 404 when following an internal link is their fault. Usually they just assume that the webmaster was sloppy.

Why not use a 503 Service Unavailable? This is telling (stretching ;-)) the truth, and people do assume that a 503 is temporary, because that's the explicit point of the message. And you can throw in an e-mail link, like someone suggesed, if they want to be notified when the service is "back up".

Or I suppose you could use 501 Not Implemented if you really want to be honest, but that's boring. ;-)

ajkirwin
Usually when I see a 503, I think, "Great. The site is f-ing broken/overloaded/down" again.

A 404.. I find, not so much.

simplegeek
Why not simply be honest and tell them "Hey, we intend to introduce these features and testing few ideas. So far, 445 people have expressed their interests, including you. Thank you for your interest and time. You can enter your email below (this is optional)and we will let you know as soon as this is available. Thanks again."

Two birds in one shot :)

litewulf
I think there is something stunningly wrong with assuming your users are idiots.
kirubakaran
>... most of the time the visitors to the site blame the 404 on their own browser, connection, or service provider ...
breck
That was my impression a few years ago.

But it's more than a solid idea. It's a proven technique. It's in use in more than a few major websites(I know from experience).

The beauty of it is: A) Couldn't be any simpler and faster to implement

B) A 404 page actually can be a better user experience. If you give the user a "Coming Soon" or "Under construction" or "We didnt really build this page, but wanted to AB test whether it would be worth it"---those can be actually more confusing to the average internet user than a 404. Remember, the average internet user probably doesn't even know what market testing means.

C) In my experience, when it is used it's used in a very small fraction of visits(just enough to get a relatively significant result set), so only a very small number of visitors actually reach the 404.

D) The users that reach the 404 might be annoyed for a sec, but generally return to what they were doing before.

E) Because it's so fast and easy to implement, and there is only 1 key variable to look at (number of clicks), it can quickly be used over and over for many different situations(think: easy to automate).

Rexxar
I think a small "beta" icon near the link could be a good thing too : It prepares the user to have an error message.

less surprise ==> less disappointment

jmtulloss
I don't think so. People might look at that and think "Oh, this isn't ready yet." While that might be true, it also destroys your test.
[deleted]
(comment deleted)
haasted
Any guesses as to the name of the unnamed "wildly popular web site"? :)
swombat
Twitter.

They also do these tests regularly to make sure we still need existing features.

jonknee
Twitter 404s for a whole other set of reasons. But this site isn't Twitter, because it has been acquired:

> ... and it was acquired not so long ago for a figure with 9 digits

s3graham
Ah, but they use 503, not 404.
randomwalker
Could be powerset. Three clues:

1. Recently acquired by MS for a rumored 100M.

2. CEO is talking about 404 pages. Powerset's CEO is indeed a tech guy, Barney Pell.

3. "Thousands" of visitors per day sounds about right for powerset.

[dead] breck
I'm pretty sure I know it. It's not Twitter or Powerset.
keefe
"But upon further thought, the 404 test is brilliant. Why? Well as the CEO aptly put it, most of the time the visitors to the site blame the 404 on their own browser, connection, or service provider anyway."

I think this is a faulty argument, since you know most users will hit refresh and realize it is your site after about 30 seconds? People are pretty familiar with this new fangled web browsing on those interwebz now...

timcederman
This is terrible. You could at least put up a custom page, with perhaps an email field offering to notify the user when that feature is available.
grhino
Anybody else have a problem with the fact that visitors will blame their computer or their ISP first before the website that returned the 404 error?

My personal experience is that if users are confused about what system to blame, they blame any systems they can!

I'd guess this approach works if you have a large user base with a lot of churn. It doesn't matter if you upset a particular user because they probably wouldn't return anyways.

truebosko
I'm sorry but there are many, many other ways to test if people are interested in something. This is a horrible idea.

Poll users, send out a newsletter and see the response, post a link to a page telling them what's coming and see how many click. If no one clicks, then no one will miss out on this new upcoming feature that all of a sudden isn't coming.

andreyf
The author is making the assumption that clicking on the 404 link is correlated with interest in the feature... I'm not sure if that's always the case.
13ren
I agree it's not a perfect correlation, but there would be some, most of the time.
babul
On a lot of sites, especially those with a set navigation layout (where a new link/button is more obvious) and regular user-base, users will often click on a new link/button just to see what it is and not always because they want it.

I cannot help but think this is only useful on (high-volume) sites where individual users are of low value i.e. do not pay to use service or contribute much, and for anything more sophisticated or with savvy users it would be deterimental (imagine seeing this on the stock-market web-app you pay $$$ for).

It all depends on how often you use this trick really.

mechanical_fish
I think it's almost always a bad idea to create a subconscious association between your brand and something that doesn't work properly.

Yeah, maybe the rational user will conclude that the 404 was somehow their ISP's fault (although I, for my part, tend to regard 404s as a sign that either the link or the site has aged out of existence). But convincing the user's conscious mind is only part of the battle. At the subconscious level, you've just lost: The user now associates your site with brittleness and broken promises.

I think it's probably better to test with a "beta" page, or a "coming soon!" mailing list, or something other than a metaphorical brick wall.

babul
"Fool me once Shame on you. Fool me twice Shame on me." --Chinese Proverb.
jodrellblank
"Fool me once, shame on ... {404 proverb not found} shame on you. Fool me, can't get fooled again"
utnick
I read about a similar idea ( in 4hww maybe? )

Basically take out adwords ads for products you havent developed yet, see which ones get clicked, develop those products.

sh1mmer
Lots of people only look at a feature once out of curiosity. Feature usage is a different thing. Just noticing something is there might cause me to click. Whether I would continue to use it would totally depend.

All this tracks is how many would try out a feature at least once which isn't the same as how many people actually want it.

jodrellblank
Heck, with this we can completely solve "build something people want" and fill in the ever mysterious step 2 into the bargain:

1) Create a home page peppered with links (all 404s)

2) Fill in the links, prioritising by link click count, building your company into one which solely and only does Things People Want (tm).

3) Profit

[deleted]
(comment deleted)
hc
this seems like a really boneheaded idea considering that search engine ranking algorithms penalize broken links
thomasmallen
Or you could put a simple tracker on the page...I'm surprised that someone there couldn't think of a better approach to web stats than Apache error logs...