HN Explorer

Tell HN: How to ban url shorteners from linking to your site

22 points by pj ↗ HN
This is just an idea I'm going to throw out there. It seems a bit odd to suggest it because any link is a good link, right? Maybe not.

Anyway, something like this in a bit of javascript would prevent people from linking to your site using a url shortener:

  <script>
  if (document.referrer.indexOf('bit.ly') > 0 
    || document.referrer.indexOf('someothershortener')>0)
   document.location.href='http://yoursite.com/noshortener.html';
  </script>
Put in that noshortener.html some content that will tell people why you don't allow shorteners to your content, here are some suggestions:

  1) You want to know where your visitors are coming 
  from and shorteners steal the real referrer.
  2) Shorteners are a layer of indirection that could 
  be modified beyond the user of the shortener's 
  discretion to do something like digg is doing now
  3) If the shortener goes away, the link to the 
  content is no longer valid
  4) It slows down the web due to one more DNS lookup, 
  one more server redirect, etc...
Most users of shorteners, when they use one to link to a site are going to click the link just to make sure it works. When they do, they'll see your banned url shorteners page and it will help all of us "experts" in the web who understand why url shorteners are bad educate the lay users of the web as to why those url shorteners are bad.

Yes, we take a hit, but we are sacrificing a little traffic for a better web experience in the future.

42 comments

[ 2.9 ms ] story [ 94.1 ms ] thread
a2tech
I tend to think that any link and visit is a good one. What I would prefer to do is using the above javascript, detect incoming traffic from URL shorteners and display my own informational bar across the top with the pages real URL and perhaps simple stats of how many people were referred to that page using a shortened link.
pj
That's a good idea, instead of

  document.location.href="...";
You could use:

  document.getElementById('urlshorteners').style.display='block';
Where originally, the div is like this

  <body>
  <div id='urlshorteners' style='display:none;'>
        Url shorteners information...
  </div>
I like that idea.
jerf
Honest question: Why do you care?

Consider this like an impromptu poll about why you care about not getting linked via a shortener. (If you don't care, I'd ask that you not post a reply to my message here, though I obviously can't enforce that.) I ask because I wouldn't care, but I am interested with an open mind in why someone would.

calambrac
See current DiggBar scandal. These shorteners put a middleman in between your links and your content, allowing them to, at any point they like, do whatever they want, including sending all those links to themselves, to a competitor, to a landing page strewn with ads with your content framed inside, whatever. As much as we all love middlemen in other industries, seems like a great idea to let them in here, doesn't it?

Joshua Schachter wrote the definitive anti-shortener argument:

http://joshua.schachter.org/2009/04/on-url-shorteners.html

pj
I think really, because I care about the future of the web. I know there are myriad problems with url shorteners and they are jeopardizing the future of the web. My website is just one little piece of it, I know, but education is grand. I think people, when presented with rational reasons that benefit us all, will listen to them.

The person linking to your content is interested more than anything in bringing users to your content. If you put up a page that prevents them from doing that, then they'll use another link, or not link to you at all.

Really, it's about education and education comes at a cost. If you are educating, then you aren't focusing on coding or something else.

On the page where the javascript redirects, you could say, "Thank you very much for linking to my content. I think it's great that you appreciate what I have written, but please understand that the method you are using to link to it is not good and here's why: yada yada yada..."

ubernostrum
You really think that being able to get a shortened URL to put in a post on Twitter is such a threat to the future of the Web that you need to, essentially, implement a huge "fuck you for trying to visit my site" message?

If that's the case (and believe me, I've read Schachter's bit, and many of the zillions of "me too" posts following it, and so know the arguments inside and out at this point), I really don't have anything to say to you except "congratulations on your new job in the newspaper industry, I'm sure you'll fit right in".

joshu
Indeed.

I guess the main points were:

1) The shortener ought to be owned by the publisher; at least then the document and the identifier are connected within the same organization/entity/whatever. At least that way if the URL is lost, the document probably is too.

2) The shortening is necessary due to design decisions by twitter (and poor ones, at that.) Changing these in the right way could make the entire problem dissapear.

Punishing users is probably not the right way.

ubernostrum
Well, URL shortening has value outside of Twitter. There are plenty of services which generate absolutely enormous URLs which would be a pain to sling around on their own, and being able to get a quick, short version to refer to is awfully handy (think "directions on Google Maps").
calambrac
And before Twitter, that's the only kind of thing these services were used for. They were hardly prevalent, and they certainly weren't "necessary" in any sense. The downside just didn't really matter that much.

Now that's starting to change. By analogy: there's a legitimate use for DDT, and when I go camping once a year I happily buy a bottle of it, but that doesn't make spraying it out of a firehose around cities a good idea.

ubernostrum
You're blowing this so far out of proportion that I'm not sure it's possible to have a reasonable discussion with you.
calambrac
You really believe that people who think heavy widespread use of url shorteners is bad for the web are too unreasonable to even speak to? Dude... alright, whatever.
ubernostrum
You compared the use of shortened URLs to spraying poison out of a firehose. Can't quite put my finger on it, but yeah, I think that's a bit unreasonable.
calambrac
You do know that they did actually used to spray DDT out of a firehose, right? That that wasn't just some random bullshit I made up on the spot, it's actually a case where we did wrong by taking something that's okay in moderation and trying to use it in large quantities? Do I really think url shorteners are as bad as spraying poison out of a firehose? No, but it was a fucking analogy.
zain
None of the reasons you listed are significant to the person reading them. It is not the visitor's problem that you can't track referrers or that it slows down the web, and the fact that they're seeing this page means the shortened is working as intended. Furthermore, you're basically making it impossible to share links to your site on Twitter (where they get auto-shortened).

All in all, you're doing little more than pissing off visitors you've probably worked pretty hard to get.

pj
the fact that they're seeing this page means the shortened is working as intended

They won't actually see the page they used the URL shortener to create a link to, because the browser will automatically redirect to another page where you have explained why you are disallowing url shortened links to your content.

I also disagree with your premise. I think the reasons I mentioned are significant to all users of the web.

I do understand the issue you have presented though, that the short term benefits to the user of using a shortened url, may outweigh the long term ramifications I have listed, but perhaps as a group here, we can come up with some arguments that may help us content producers take back the web from the url shorteners.

Really, I just think when taken to the logical conclusion, a web saturated with url shortened links is really a not very good web.

I'm trying to propose a solution here, it may not be the best one and it will take a coordinated effort among content producers. I suppose those who ignore the call to arms are going to "please" url shortener users more than those who don't, but with a little effort and a genuine interest in a sustainable web, I think we can win this war.

rex
...the browser will automatically redirect to another page where you have explained why you are disallowing url shortened links to your content.

If you're going to the trouble of serving up content from your site, why would you disallow the content they came to see anyway? Maybe the concept could work if you had support from higher-up in the redirection process (from url shortening services themselves), but to effectively re-direct from your own page to another "you an't access this content" page on your site will only make end users confused and disinterested, driving traffic away.

I agree that URL shorteners are bad and acknowledge you're trying to do something about it, but this isn't the way to solve the problem.

calambrac
Because it's more important that the shorteners die before they become more entrenched than that I get their traffic. The fact that I can seriously say this right now is something worth fighting to preserve.
ubernostrum
TinyURL has lasted longer than many of the sites its links point to.
calambrac
The fact that for however many years TinyURL served a niche purpose hiding goatse links somehow means the recent explosion in the use of these services is positive for the web? And: what good is a shortened link to a dead site?
ubernostrum
"And: what good is a shortened link to a dead site?"

About as much good as an un-shortened link to a dead site, which is what it turns into after the redirect.

I guess my problem here is I still haven't seen an actual argument to back up the sorts of "behold a pale horse: and his name that sat on him was Digg, and Twitter followed with him" hysteria that's been going around. Yeah, sometimes links die of old age; we've been dealing with that for years and it's not a new or unique problem. And yeah, sometimes people hide nasty things behind redirects; again, we've been dealing with that for years and it's not a new or unique problem.

So what's with the gloom and doom?

calambrac
About as much good as an un-shortened link to a dead site, which is what it turns into after the redirect.

That's exactly my point; you were the one who pointed to TinyURL outlasting some of the sites it stores links to as though that were a good thing.

Yeah, sometimes links die of old age; we've been dealing with that for years and it's not a new or unique problem.

Yes, but this introduces another failure mode, where the end target could be fine, but it could be the shortener service that dies. That sure is awesome if it's your content being linked, and you have no way to fix it, huh? At least with regular links, if a link goes dead, it was your own fault, and you can fix it if you want.

And yeah, sometimes people hide nasty things behind redirects

So why would you want to create more opportunities for this to happen?

We've been dealing with that for years and it's not a new or unique problem

But not on the scale that we're starting to see now. You seem to be operating on the premise that things we've seen used in moderation without too much trouble are automatically safe for heavy use, but the Digg thing shows how ripe for abuse these services really are.

callmeed
I'm all for blocking framing sites like the Diggbar, but killing all URL shorteners is a bad idea, IMO.

With Twitter growing so fast and automatically shortening pasted URLs, you could actually be losing some quality traffic if people start talking about your business/app/site.

calambrac
Someone with a big vested interest in seeing the internet not break needs to step up and provide a shortening service that says "if you own the target, you own the shortened link, period".
chaosprophet
I agree. Most of my sites have frame busting code in them, simply because most of the times, the frame spoils the look of my page. Also the frame sort of implies as though my page is part of the site that does the framing, and I find that annoying.

As regards URL shorteners, I don't mind them one bit. The only problem I have with URL shorteners is that they may be used to disguise malicious links, and even that problem is being solved (bit.ly provides automatic warnings).

philfreo
I think if you're going to do this it'd be better done as a server side redirect, like through .htaccess

However even though I'm, in general, against URL shorteners, I wouldn't do this because I think there are some situations where they are helpful.

pj
That's true. I just threw this together. I think you could even create a database table and generate the redirect on the server side or dynamically create the javascript. Most browsers have javascript enabled. Maybe a hacker with some time (maybe I'll create one) could create a site with a dynamically created .js file and a remote link like this:

   <script src="http://takebackthewebfromurlshorteners.com/bannedreferrerslist.js">
  </script>
This .js could be dynamically generated from a database table of known url shorteners. Such a service could even get really fancy and show stats from all the known shorteners, etc.

You could add parameters like this:

  <script src=".../bannedlist.aspx?allow=digg.com,bit.ly,etc..."></script>
Then the user could customize exactly which url shorteners they want to allow.

All kinds of things a service like this could do!

tokenadult
How long are the URLs on your site, that they are getting shortened so often?
andrewf
Putting as much content in the URL as possible is a common SEO tactic.

A bit silly, really - we're producing readable English links targeted at the machines, and short strings of unintelligible gibberish for humans to look at and click on.

jrockway
Putting as much content in the URL as possible is a common SEO tactic.

Are there any controlled studies that shows that this does anything useful? Or is it an imagined effect? ("We think it should work this way, so it does!")

paul9290
ALl links will die someday! Not every website is going to be around forever!

The only ppl who hate on URL shorteners is this crowd here while millions of others usual them daily!

We are hackers and thus if this is such a big issue and change, one of us should do something about it. Something that works for the users and solves the issue about 3% of all users rant about!

tallanvor
Why not pass the url the user was trying to visit to your noshortener page? Then you can have a brief blurb and let them click the link to continue. You still don't get full referrer information, but it's better than nothing and the user gets to view the content.
keyist
The reasons you give are all valid reasons. However, you miss the reason most likely to sway end-users: a shortener service can relate every page a user has seen to that user (or at least, to his ip). The privacy angle is the biggest factor imo.

In your info page you may also want to promote Untiny (http://untiny.com/extra/) as a service which helps bypass shorteners with minimum fuss.

ErrantX
As a user if you did that to me I would just close the link and lose interest in your site....

Yes, agreed, the DiggBar style thing IS annoying and we should stop that. But short links are important.

Several forums I frequent insist you use shortners for URL's longer than 25 characters (you get a warning if you dont!) to keep things neat. Twitter - your isntantly killing traffic from that crowd.

Now, showing a box on the page with some of that info would be useful / understandable.

jasonkester
While we're talking about it, here's a feature I'd like to see these URL Shortening services impliment:

- If the "Shortened" URL you are about to give me is actually longer than the URL I gave you, just give me back that URL.

I see this sort of thing on Twitter all the time, and it drives me nuts. "hey, check out:

http://tinyurl.com/mweytn

... which redirects you to...

http://twiddla.com/

Uh... What exactly have you shortened again?

skwaddar
I have no idea what your beef is, I think you have a stick up your ass about something.

Why you think you're some sort of web "expert" is beyond me. A real expert wouldn't be using javascript for this for a start.

Punishing third parties because of some moral crusade is dumb.

Slows down the web hahahah jeesh.

throw_away
it would be nice if url shorteners did a double redirect so that the following happened:

1) user clicks on http://sho.rt/abcdef located on http://original-referrer.com

2) sho.rt redirects to http://sho.rt/abcdef?original-referrer=http://original-refer...

3) second url redirects to the content.

in this way, content owners will know where their traffic is coming from, people who place shortened urls don't have to do anything special and the pasted urls remain short. the only downside is a slight increase to latency to retrieve the second redirect.