I understand closing git.io if they were having to spend too much time moderating it, but they could have limited it to links to github.com instead. We use git.io a lot on irc because channel topics have a short character limit and links to pages on github wikis/etc are useful.
Honestly I have no idea. But if they were having issues with spam hosted on github.com, what problems does deactivating git.io solve? The spam still exists and it's hosted using their service. Just the link is a bit longer.
If you've missed the news (and wondering why this is being voted), initially GitHub plans to fully shut down Git.io without redirection. Understandably, many people were dismayed by that and concerned of brooken links in research articles, printed media and source code comments (https://news.ycombinator.com/item?id=31162829).
As a researcher that has used git.io in at least one paper in the past, this is a relief. Unfortunately, GitHub permalinks to specific files (at a specific commit) are rather ugly, especially if you need to put them on paper. Hope they come up with an alternative.
That's even worse than using some larger url shortener used by many people. The incentive to keep that one running are way larger than having it as side project of some publisher or university.
What happens when the shortened version becomes ambiguous?
For instance that's commit 9e02977bfad006af328add9434c8bffa40e053bb.
What happens when someone creates a commit and that happens to have 9e02977bfa-e-something?
I'm not even sure what github does here. Does it just refuse to select a commit or give you a list of candidates and you have to check the date (given that your link presumably was unambiguous when you wrote it)?
You're right that the chance that two specific 10 digit hashes collide is 1 in a trillion.
But the chance of a collision between a specific commit and any other commit goes up linearly, so for the Linux repository it's about 1 million / 1 trillion so 1 in a million.
The chance of a collision between a any two commits goes up quadratically (while it's small), and becomes ~50% when you reach the square-root of 1 trillion, i.e. 1 million. So there is a good chance the Linux repository already contains colliding 10 digit hashes, and if they don't exist yet, it'll likely happen during the next couple of years.
Yes, there is a good chance there is already a 10 digit hash collision in the Linux repository.
But if you pick an arbitrary commit to cite in a research paper, it's still 1 in a million that that particular commit has a collision. And this is for a repo which has to be in the 99.999th percentile of number of commits.
You can do the math and calculate risk of collisions. And then put that in relation to the context: It is a single repository, probably with very few commits since it is research and not a highly active commercial product.
You could probably cut the commit hash to 4 characters and still not have collisions for this application (ie. 1 / 65536 chance of collision @ 4 characters), or even fewer
That's assuming researches would only link to research repositories, instead of doing research on actual applications - including massive things like the linux kernel.
GitHub only parses commit hashes with at least 7 characters. IIRC I read somewhere that this was considered a safe lower boundary but my googlefu couldn't come up with a site that shows the math.
The chances of a hash collision by chance with 10 characters are basically impossible (unless sha1 is broken further and it's a malicious repository).
The master branch of the linux kernel has a little bit over 1 million commits (measured with git log --format=oneline | wc -l).
With "git log --format=oneline | cut -b 1-4 | uniq -c | sort -n | tail -n30" you can verify that it has 25 prefix collisions with a 4 character prefix. With 5 it's still 4 collisions, with 6 there are none.
>With "git log --format=oneline | cut -b 1-4 | uniq -c | sort -n | tail -n30" you can verify that it has 25 prefix collisions with a 4 character prefix. With 5 it's still 4 collisions, with 6 there are none.
I'm sorry to say, your script doesn't work. You need to `sort` before `uniq` as that only counts adjacent duplicates.
The linux kernel as of 5.12-rc1 had 25 prefix collisions for the prefix "ffeb" alone (and 33 for "e3f2"). Here are the full shas:
You are completely right. I don't know how I missed the needed sort call.
It's even more stupid considering that a 4 character hash has only 16*4 => 65536 different strings so there must be a lot of collisions in over a million different hashes.
A tag can be changed. Barring the extremely unlikely chance of a (non-malicious) hash collision, the commit hash will always identify the specific commit intended by the referrer.
As commit hashes ought to be globally unique—the same hash can appear in more than one repo but they will all have the same content and history—GitHub could offer a shorter form of the URL which omits the user and repo names, in exchange for a longer minimum hash length. This would require creating an index of all the commits reachable from public repos, unless they already track that.
Sure, maybe "ugly" is not the best term here. The fact is, the "short" URL you provided is still 74 characters long, while a line in a well-typeset paper is usually 70-80 charecters long. Plus, username and repo in the example are pretty short ("torvalds"+"linux"=13 characters), which is not always the case. So these URLs are definitely hard to put on paper in an aesthetically pleasing way.
> There's all info in the url that you even can use in the future if GitHub ever goes away.
GitHub could maintain a .git_io file (or whatever) in the repository with the short->long URL mapping, for instance. So, the info would survive GitHub.
Apologies if this is completely naive about how academia works, but can’t you just use a clickable link in the paper? Is it really so common to print a paper out and then manually type the URL back into a computer?
No need to apologize :) But it is frowned upon. An article is a textual, rather than hypertextual, document. Links can be used to make the PDF easier to navigate, but should not make the printed document inferior. So, in short:
* The file is available at the following URL: http://example.com <- good
* The file is available here <- bad [here = clickable link]
Huh, someone should make a lookup table of git.io URL to actual URLs, one "someone" I can think of is archive.org, who would again have to clean up after people messing up the Internet by just abandoning shit.
And then someone else can make a browser extension that automatically queries this look-up table when it sees the browser user has clicked a git.io link. It's like DNS, but different!
(Yeah the cleanest option would be for a trusted someone to take over hosting of the git.io service, but hey, welcome to the present...)
Is this Microsoft "killing" off a perfectly stable and low maintenance/overhead product/service just to save a few pennies? As a GitHub paying customer, I would gladly pay an extra few cents per month for this feature to continue. We use link shortening all the time especially for linking to a specific commit + line reference in our code. Shutting this down makes zero sense. :-(
I think the problem is that it wasn't low maintenance. The moderation of the links for malicious links was becoming an increasing drain on resources. It's the same problem most large tech services run into eventually.
Unfortunately, most people just throw "eh, it's low-maintenance".
To bring down the point, here's the source code for that link shortener: https://github.com/technoweenie/guillotine. It was last updated in 2015. If you don't believe that this is the source code, this was directly linked to in its announcement: https://github.blog/2011-11-10-git-io-github-url-shortener/. If the extreme lag (in an attempt to save the links) is indicative if its backbone, it's probably just running in a single server, which is very likely to be horribly outdated. It might not be due to the cost of removing the malicious links, it might be that Git.io as a platform has a cost in of itself (even excluding common things like domain and hosting costs).
... and you didn't understand the underlying problem, which proves my point of people quickly dismissing "low-maintenance" applications. This isn't a Windows application which still chugs because of immense herculean compatibility made by Microsoft, this application, which was supposed to be quickly installable in Heroku... isn't. That says a lot.
Very unlikely to be honest, especially that there's no private* data present in Git.io.
* Technically the links are not publicly listed, which might jeopardise some obnsscure but technically-available repository, but it doesn't store private data.
Since January 11th, 2022 you could no longer generate new git.io links (See [0]). If this feature is used "all the time", you should have gotten problems in the last 4 months.
As an example, I run an online store, Google checks the landing page urls for ads once a day. If you happen to have a temporary outage (surprisingly common on Heroku) at that exact moment your ads will be deactivated and not run for up to 24hrs!
I’m somewhat suspicious it also dumps your ad back into a learning phase.
And so, the Sun began its sullen plunge into the horizon, until a dusky pall crept over everything. Night’s drear countenance faced down the light of all that had gone before it; now fading, now flickering, now Extinguished.
What was the effect on security? Really curious about the security implications as they've shown that as the reason to shut it down almost immediately without the grace period.
Probably none, just the balance between resources needed to keep it running (more dealing with potential legal and brand issues then technical) vs shutting it down shifted in response to the backlash and bad press around shutting it.
> That said, we still encourage users to make use of one of the many URL shortening services available with greater functionality than the git.io service provided.
Unless you need to change where the URL shortened link points to, don't do this, provide the full, canonical link instead.
Is it common in papers to attach a copy of a reference as an addendum? That would make sense if the sources cited are web pages.
Thinking about it, Wikipedia should maintain their own index and archive of cited sources, instead of or in addition to relying on archive.org.
> Thinking about it, Wikipedia should maintain their own index and archive of cited sources, instead of or in addition to relying on archive.org.
Why? Wikipedia and the Internet Archive are partners, each working to their comparative advantage. If Wikipedia has archival needs, they ask the Internet Archive to facilitate those needs.
In case you're looking for an alternative: consider using Perma.cc links, which both serve as a URL shortener and an archive of the current page.
(Plus they're expressly intended to last "forever" and are maintained by one of the few organizations with the multi-century history to credibly make that claim (Havard Law School))
That's a good point. For .cc in particular, it's the TLD for an Australian territory and is managed by Verisign so I'd expect the risks to be lower than with some TLDs. Still, though, a lot can happen in "forever" and you're right that it adds an (unnecessary?) risk
That's odd, the US usually has Secretaries and Departments instead of Ministers and Ministries. Is it a cabinet level position? I'm having trouble finding it's web page.
But I would have used `.net` personally, rather than a country-specific domain.
They stopped giving out the monthly allowance of 10 links to free accounts in 2019.[1] New accounts now can archive 10 links, and that's it.
You might have access through a registered institution (e.g., libraries and universities get it for free). Otherwise, the rates for individuals are hidden on their blog.[2]
If you really want to archive and shorten a link at once, https://archive.today (often seen on HN for bypassing paywalls) is another option, but its unknown ownership makes its long-term existence slightly questionable. For general web archival, the Internet Archive (https://web.archive.org) looks like it'll be around for a little longer. But since its crawler and server have their own quirks, I'd still submit links to multiple services for good measure.
Speaking of web archive reliability, it looks like WebCite might have recently died for good, after a decade of funding issues and intermittent downtime[3][4]—the first major web archive to fall?
I’ve never heard of perma.cc before, but had a similar idea (likely more naive than however they are doing it) to save the content of a given url as an image and apply OCR, and keep both the image and the text saved. Got that working (somewhat well) and then dropped the project heh.
URL shorteners were (still are!) probably the worst idea/trend on the internet. They solved a silly short-term pain (twitter not storing full URLs) but introduced a long term dependency, and normalized tracking via these links.
DNS solved vanity memorable URLs, and continues to do so if someone is not looking for a .com
URL shorteners offering vanity urls solved people not needing to deal with the hassle of purchasing a vanity domain via a registrar, setting up DNS records, and waiting for propagation.
Are you suggesting that instead of short.com/foo you register foo.mydomain.com and setup a reverse proxy to redirect? Isnt that just a url shortener with more steps? Why wouldn't I just setup my own shortener service instead and send a `POST` request when I want a new url?
URL shorteners came later, and solved the problem of people needing to deal with “more steps” in order to have short memorable urls. Achieving memorable urls in “less steps” is one of the major selling points of url shorteners.
However, url shorteners did not solve short memorable urls — DNS had already solved that. But, as you’ve pointed out, acquiring a domain and setting up DNS records takes more work than simply using a url shortener service.
I use them to get long GitHub URLs for diffs between tags into IRC messages (which also have a line limit). It's used by cpythons commit notifier for example.
IRC line limits are normally plenty for GitHub URLs—my vague memory (haven’t used IRC for quite some years) is that 512 was the proper wire format length limit, and I can’t remember what other guff there was in transmitting a message but less than 50 characters, so it should still be over 450 characters. Think I did encounter one server with a considerably lower limit once, maybe around 200 so it was probably 256, which is still oodles for all normal GitHub URLs. (Oh sure, you can exceed it easily enough if you want to, by defining ridiculously long ref names, or referring to a very long path.)
But really, opaque short URLs are significantly worse than the full URL: you have to go to extra effort to create them, and extra effort to retrieve from them, and you’re adding an additional point of failure, and you can no longer just read the URL.
No there aren't. Do you suggest setting up nginx on one machine as a reverse proxy to the long link the second machine can access? Many ways for sure but typing out a few letters is the easiest. I do this for regular desktops but I do it more on headless machines or when installing an OS.
Firefox has a feature where, if you're logged into firefox, you can see what tabs are open on your other browsers. It makes sharing links across computers trivial.
Chrome has a similar feature when sync is enabled: the "tabs from other devices" section of the History page (Ctrl-H). The context menu for tabs also includes an option to send the tab to another device.
QR codes work for me PC -> mobile ('Create QR code for this Page'), but I haven't really pursued the other direction... I usually post to Twitter and delete it to get the t.co link now that goo.gl is done.
And would it matter if the shortener you used for that disappeared a couple years later? The stupid part was sticking a middleman into millions of links permanently posted on public websites and assuming these giant free services would keep running forever.
There are worse things that could happen than shutting down too. Someone with control of an old shortener domain could set up spoof versions of the site that a link used to forward to and use it to harvest passwords or get people to install malware.
My point is that lekevicius is correct that they're a terrible idea for long term use. If you're only using the shortened URL for a day, the downside/risk is a lot smaller.
That link looks ridiculous and isn't suited for sharing in many situations. For most people, this is the link they really want:
https://www.amazon.com/dp/PRODUCTID/
I know there are tools out there that can shorten URLs like that. I used to have a browser extension which did exactly that. It doesn't work for everything, but at least your link wont die because a third-party service shut down or messed up.
It's kind of tricky to do in general case, e.g. even hackernews is keeping meaningful semantic information in id= query parameter.
Because of that it ultimately needs to a site-specific database/algorithm, perhaps with a fallback to the default behaviour like simply cleaning up the most common garbage like (_encoding/usg/etc). I suspect it's possible to use some sort of machine learning to guess the meaningful parts of the URL path/query/fragments, but even for that we need some human curation for the training set. I wish we could collaborate on a shared database/library for that, have sketched some ideas/applications/prior art here: https://beepb00p.xyz/exobrain/projects/cannon.html
I started thinking about it since I have a similar problem in Promnesia (https://github.com/karlicoss/promnesia#readme), a knowledge management tool I'm working on. Ideally I want to normalise URLS, so they address the exact bit of information, and nothing more.
Yeah, sadly, to get the canonical attribute, you need to fetch the URL first (which is slow and wasteful). Also sometimes canonical would still be different on the desktop and mobile version of the site, so it still has to be normalised after that
I don't even think you need a machine learning algorithm. I follow a simple process all the time to achieve this.
load the page with the original URL
for each part of the URL:
remove that part of the URL
load the page with the modified URL
if the page rendered differently:
put that part back
You'd need to incorporate an ad blocker, otherwise changing ads on each reload could screw it up. Of course, you'd also probably want to hard-code in logic for popular websites like Amazon to avoid wasting time with reloads.
And I know there are limitations to the concept. It wont work for pages which require authentication or any kind of session data. It wont work for pages which are intentionally dynamic. It wont work for sites which cannot be accessed by the service. It wont be useful for pages who simply have long URLs. But it probably covers the many common use cases for a URL shortener, and you could always fall back on traditional shortening methods (redirecting) when it doesn't work.
So it basically automates detecting useful bits for a particular URL, but it's kind of time consuming and flaky. It could be very helpful to populate the 'rules' database though, and then this database could be shared with other people so they don't have to scrape.
I guess when I said ML (or preferably some fuzzy algorithm/heuristic), I was referring to generifying rules so they also work on the sites not in the rules database. If humans can detect garbage in the URL looking at a few examples, the computer can too :)
Affiliate links used to catch some heat on HN but https://amzn.to seems to be showing up in more and more comments without too much blowback. It wasn't mentioned much that affiliates could view everything people bought while under their cookie, but I'm not sure if that's still possible.
Twitter... have you ever tried to share a Sharepoint link. Those take up like 4+ lines in emails, it's a nightmare. Back in the day, Microsoft even used to have an internal url shortening service that people used to clean up Sharepoint links.
I don't understand companies using a 3rd party service for something like this. Unless you really, really need an ultrashort domain, it's pretty trivial to do your own - for example, if you have an aws account you can use a bucket as a redirector.
Yes, url shortening is about the simplest thing you can build honestly. I'm all for not hand-rolling everything but its honestly just not worth the cost/headache of relying on someone elses service for this.
On the flipside, "build it yourself because it is easy" is what led exactly to the mess at hand with git.io. It was someone's fun hackday project that quietly people and teams (including teams internal to Github) started to rely on it and then the site turned out be a maintenance burden and spam vector and needed to be shutdown.
Yeah, I'm just saying that's how it sometimes starts: a "quick and dirty" internal tool to just get something done snowballs over the years into "this needs an easier self-service interface to make it easier for Bob in Accounting because he just has so many slides" into "hey since we have this great self-service tool, let's make it customer facing and bill it as a product since we already put all this work into it".
Maybe your internal systems are less prone to such complexity effects that turn "internal only" tools into "customer facing products" accidentally all the time, of course. Just that it's an interesting risk, well worth stating, and it is fun seeing all comments on shutdown notices of exactly one such "internal only" tool that accidentally became a "customer facing product" saying "just build it yourself". "Just build it yourself" doesn't entirely protect you from it being your "internal only" tool that's the next trending HN deprecation notice headline.
The "it's pretty trivial to do your own" piece is trivial if you already have gone through the effort of setting up an AWS account and finding a globally unique bucket name across all S3 users. That option by the way requires entering a CC, personal details, and requires you to pay expensive rates compared with shorteners which are typically free. Not to mention that the resultant URL is still comparatively long
Most people won't do that.
FWIW, most larger companies I've worked at have their own internal URL shortener that they run for all their internal docs and whatnot. That doesn't solve the public use case though.
Yes, I have seen that and I knew someone would reference it
Sure, most people won't do that. But the user base of github is not most people, it's developers, a large fraction of which work at a company which has an AWS account or equivalent, and I questioned specifically "companies" not "people" using an external service for this.
And the comments elsewhere on this thread should remind us all that "a lot can happen in forever"... My interpretation: eventually, everything will be deprecated. There may be some web sites (github.com, for example) that are "too big" (or "too important") to fail or disappear; but they eventually will.
It might remind us hyrum's law, but in this case weren't the people who are not wanting it to go away -- using it as intended?
I feel like there's a different "law" here, involving persistence and memory and history and recordability on the internet -- people want it, but it doesn't come for free, and there isn't necessarily anyone with a profit motive or business plan to make it happen.
"With a sufficient number of users of an API,
it does not matter what you promise in the contract:
all observable behaviors of your system
will be depended on by somebody."
URL shorteners still serve a purpose for sms, marketing, smart redirects, and more. One of the ways to combat some of the concern around what happens if the service shutdowns is to use your own domain when creating short links. This allows you to export your links since you control the domain.
I've been running https://T.LY URL Shortener for the past 2 years and I'm the creator of the URL Shortener Extension (https://t.ly/extension) that has over 400K users. Let me know if any questions around url shorteners.
My biggest problem with short URLs is they appear spammy. I can't look at it and decide whether it's safe.
Just yesterday I received a text from a shortcode with a "bby.me/XXXX" URL saying my Best Buy order was canceled. Turns out it was a legit order I placed nearly 6 months ago and was on backorder that I forgot about.
I'd love for a shortening service to offer an "unfurl" tool where I could paste in the URL, and it shows me what the redirected site is without clicking it and potentially activating their clickthrough tracking.
> I'd love for a shortening service to offer an "unfurl" tool where I could paste in the URL, and it shows me what the redirected site is without clicking it and potentially activating their clickthrough tracking.
For Bitly, at least, you can add a plus sign to the end of a short URL to preview what the destination URL is. I don't know if other URL shorteners have something similar.
>That said, we still encourage users to make use of one of the many URL shortening services available with greater functionality than the git.io service provided. GitHub support will not be able to update or edit redirection records served by the git.io archive service.
No, wrong. Don't use a URL shortening service. Don't use any of them. Especially not in anything long-term.
How is domain registration more daunting than the process to get a phone number? I felt like I had to give a lot more info, time and money to my phone carrier than to Hover.
I should have been more specific. Registering is indeed pretty easy. But using a domain is quite difficult without a lot of background knowledge. Everyone understands how phone numbers work well enough to use them, and there are lots of systems in place to walk you through the process whenever you need to change phones (or numbers), etc.
Connecting a domain name to a piece of software running on any of your devices should require nothing more than a quick OAuth flow to your domain provider to connect a tunnel to the application. You shouldn't need to understand DNS, TLS, HTTP, ports, ip addresses, NAT, CGNAT, etc in order to self-host simple programs like URL shorteners quickly and securely.
It can? I trust my $12/yr domain to stick around longer than a hypothetical $12/yr subscription to a SaaS link shortening service. The only exception I think is bit.ly who seem to have staying power because they're actively fighting spam and have B2B revenue.
I don't think registration is any kind of barrier at all. I think I could go from "I want this domain" to "I own this domain" in less than 30 seconds, most of which is spent logging in with my 2FA code. It's the same process as buying something on Amazon. I can't imagine how that could be much easier. The difficulty is later in the process when you have to actually host something there.
You literally type in the domain on something like namecheap.com if it says available you click add to cart and click checkout. Familiarity with the process of buying something online is the only requisite. Everything after is the hard part.
How would being able to get a domain name more easily make this problem go away? You would also need to have it point somewhere, and setup redirects if you are trying to link to places you don't own.
Sorry, I was a little vague. The implication is that if you have your own domain you can host your own URL shortener, which will never be deprecated. But obviously there are a lot of other problems that need to be solved to make self-hosting things like this easier. That's what my personal side projects are currently focused on.
I'd like to see link shortening somehow supported as git objects. That's primarily where I will always want to shorten links- to make unwieldy URLs fit in a comment.
Honestly, any kind of linking is broken in long-term...
Sometimes the content on the link doesn't exist anymore, sometimes the eg. newssite changed their url scheme, so the content still exists, but not on the same url, sometimes the media changed their story, so linking to the "same" article, now paints a different picture, and of course, sometimes old domains get hijacked by spammers... and more related, sometimes everything works except the url shortener.
Honestly, if I ever link anything that will be read after a long time, I only use archive.org/ links, and I cross my finger, that the snapshot will still exist in X years.
> Honestly, if I ever link anything that will be read after a long time, I only use archive.org/ links, and I cross my finger, that the snapshot will still exist in X years.
I don't understand this practice. It seems more proper to use the original URL and let readers themselves use archive.org when the link becomes dead.
Using the nonprofit's archives when you don't need them seems abusive. On average it should be simpler for the original server to render the page than for the wayback machine to fetch a particular snapshot. What's worse is that I don't think you're the only one needlessly using their archive like this, and wonder if, as a collective, uses like this significantly increase their operational costs.
The problem with direct linking is, that content is sometimes changed without any indication.
Something happens, you link to cnn/fox/whatever, and discuss the news story or their comment or whatever, and two hours later, the whole article is rewritten, the headline is different and the subtone is different. This would be ok, if there was a "history" section at the beginning of the article, saying "this article was changed on date X and Y and Z, click here to see previous versions", but usually it's not.
Now if i said that ny times is using those attacs for politics, and linked the article directly, you (visiting the second version of the article) wouldn't find the headline "problematic", and you'd think i was just some far-whatever nutjob conspiracy theorist... by linking to the archive, you see the headline that I saw.
166 comments
[ 2.8 ms ] story [ 237 ms ] threadIsn't this how it worked?
Which is the stated reason why they're killing it off. It was being abused and Github/Microsoft didn't want to take the time to maintain/moderate it.
For example https://github.com/torvalds/linux/blob/9e02977bfa/kernel/dma...
I don't find this ugly, it's even very human readable. There's all info in the url that you even can use in the future if GitHub ever goes away.
Actually, I would use this argument that people should refrain from using url shorteners in papers for this very reason: It is anti-reproducible.
At some point I did a fairly automated from Apache but otherwise the redirects have been zero maintenance.
Notable, they I have broken since I own them!
For instance that's commit 9e02977bfad006af328add9434c8bffa40e053bb.
What happens when someone creates a commit and that happens to have 9e02977bfa-e-something?
I'm not even sure what github does here. Does it just refuse to select a commit or give you a list of candidates and you have to check the date (given that your link presumably was unambiguous when you wrote it)?
But the chance of a collision between a specific commit and any other commit goes up linearly, so for the Linux repository it's about 1 million / 1 trillion so 1 in a million.
The chance of a collision between a any two commits goes up quadratically (while it's small), and becomes ~50% when you reach the square-root of 1 trillion, i.e. 1 million. So there is a good chance the Linux repository already contains colliding 10 digit hashes, and if they don't exist yet, it'll likely happen during the next couple of years.
https://en.wikipedia.org/wiki/Birthday_problem
But if you pick an arbitrary commit to cite in a research paper, it's still 1 in a million that that particular commit has a collision. And this is for a repo which has to be in the 99.999th percentile of number of commits.
By now it's up to at least 7 (going by my clone of 5.12-rc1 that I had lying around), and it's becoming more likely.
Those collisions happened with objects other than commits (of which there are more), but that's by no means guaranteed.
You could probably cut the commit hash to 4 characters and still not have collisions for this application (ie. 1 / 65536 chance of collision @ 4 characters), or even fewer
The chances of a hash collision by chance with 10 characters are basically impossible (unless sha1 is broken further and it's a malicious repository).
The master branch of the linux kernel has a little bit over 1 million commits (measured with git log --format=oneline | wc -l).
With "git log --format=oneline | cut -b 1-4 | uniq -c | sort -n | tail -n30" you can verify that it has 25 prefix collisions with a 4 character prefix. With 5 it's still 4 collisions, with 6 there are none.
The master branch is irrelevant - github links to commits don't include the branch (and the way git works, commits don't really have "a branch"):
https://github.com/torvalds/linux/commit/ac632c504d0b881d7cf...
>With "git log --format=oneline | cut -b 1-4 | uniq -c | sort -n | tail -n30" you can verify that it has 25 prefix collisions with a 4 character prefix. With 5 it's still 4 collisions, with 6 there are none.
I'm sorry to say, your script doesn't work. You need to `sort` before `uniq` as that only counts adjacent duplicates.
The linux kernel as of 5.12-rc1 had 25 prefix collisions for the prefix "ffeb" alone (and 33 for "e3f2"). Here are the full shas:
ffeb03cfe2b49b73da7b325a31714003761fc6d5 ffebecd9d49542046c5ecbb410af01e016636e19 ffeb1e9e897b8d36b197275592d121c96d3bdb95 ffebbecaaa86f7cde4a6a813bed14f9d56e7c373 ffeb595d84811dde16a28b33d8a7cf26d51d51b3 ffebbaedc8616cffe648202e364dce6a045d65a2 ffebe74b7c95a41d2d0ac70a44d410e0efa37ad8 ffebc8c0344934db710afc76e3bfda11a823bb3d ffebb83b34f843aadcbd3e03c4e449da14d0870d ffeb6437f018f072a330ad5911036fd020b35ac3 ffeb883e5662e94b14948078e85812261277ad67 ffebf5f391dfa9da3e086abad3eef7d3e5300249 ffebfc364dcaa5dea1a589d42207834b028df789 ffeb13aab68e2d0082cbb147dc765beb092f83f4 ffebeb46dd34736c90ffbca1ccb0bef8f4827c44 ffeb40515971c9860ff671bb074689db15e18831 ffebad7948ee0e9c619ae6e87d99437d907fc7e3 ffeb501c6cba803eefc46b570feccffe61a6d883 ffeb33d20c6217bb8f0ab46d3f1396021c00c24f ffeb80fc30acbf6bd51cb47a1815f621a9d017dc ffeb414a59291d5891f09727beb793c109f19f08 ffebedb7ab3f7964a70a1771547b26af38a189d2 ffebabe0bf0de9ee500d4605d6acb71e1ee3b79f ffeb9ec72e18e16d0b0835d959cdf01650758638 ffeb874b2b893aea7d10b0b088e06a7b1ded2a3e
with 5 characters there are 8 for b91e1.
With 6 characters there are 5 for 120bda:
120bdafaece72056e48d97809c5abe172824a7f6 120bdac7376a36418eb1d55e0161dc0e660a45c3 120bdaa47cdd1ca37ce938c888bb08e33e6181a8 120bda35ff8514c937dac6d4e5c7dc6c01c699ac 120bda20c6f64b32e8bfbdd7b34feafaa5f5332e
and a total of 28606 colliding 6-digit prefixes.
There are 9 colliding 9-digit prefixes and no 10-digit prefix.
(found with `git rev-list --all --no-abbrev-commit | sort | cut -b 1-4 | uniq -c` and variations on that theme)
It's even more stupid considering that a 4 character hash has only 16*4 => 65536 different strings so there must be a lot of collisions in over a million different hashes.
As commit hashes ought to be globally unique—the same hash can appear in more than one repo but they will all have the same content and history—GitHub could offer a shorter form of the URL which omits the user and repo names, in exchange for a longer minimum hash length. This would require creating an index of all the commits reachable from public repos, unless they already track that.
> There's all info in the url that you even can use in the future if GitHub ever goes away.
GitHub could maintain a .git_io file (or whatever) in the repository with the short->long URL mapping, for instance. So, the info would survive GitHub.
* The file is available at the following URL: http://example.com <- good
* The file is available here <- bad [here = clickable link]
And then someone else can make a browser extension that automatically queries this look-up table when it sees the browser user has clicked a git.io link. It's like DNS, but different!
(Yeah the cleanest option would be for a trusted someone to take over hosting of the git.io service, but hey, welcome to the present...)
So perhaps it's open to git.io/your-bank-here-etc
Does it support github pages? Because then it's clearly a massive problem.
To bring down the point, here's the source code for that link shortener: https://github.com/technoweenie/guillotine. It was last updated in 2015. If you don't believe that this is the source code, this was directly linked to in its announcement: https://github.blog/2011-11-10-git-io-github-url-shortener/. If the extreme lag (in an attempt to save the links) is indicative if its backbone, it's probably just running in a single server, which is very likely to be horribly outdated. It might not be due to the cost of removing the malicious links, it might be that Git.io as a platform has a cost in of itself (even excluding common things like domain and hosting costs).
You just proved the point.
https://github.blog/2022-04-15-security-alert-stolen-oauth-u...
“The applications maintained by these integrators were used by GitHub users, including GitHub itself.”
* Technically the links are not publicly listed, which might jeopardise some obnsscure but technically-available repository, but it doesn't store private data.
From the original post: “due to the security of the links redirected with the current git.io infrastructure”
What does “security of the links” even mean? Disclosure? Tampering?
[0] https://github.blog/changelog/2022-01-11-git-io-no-longer-ac...
> As we continue our analysis, we may remove individual links that point to spammy, malicious or 404 links.
Please be careful with 404 errors. They can be the result of temporary server misconfiguration.
But one could set up an html on github pages that redirects to the 'real' URL.
As an example, I run an online store, Google checks the landing page urls for ads once a day. If you happen to have a temporary outage (surprisingly common on Heroku) at that exact moment your ads will be deactivated and not run for up to 24hrs!
I’m somewhat suspicious it also dumps your ad back into a learning phase.
Unless you need to change where the URL shortened link points to, don't do this, provide the full, canonical link instead.
Is it common in papers to attach a copy of a reference as an addendum? That would make sense if the sources cited are web pages.
Thinking about it, Wikipedia should maintain their own index and archive of cited sources, instead of or in addition to relying on archive.org.
Why? Wikipedia and the Internet Archive are partners, each working to their comparative advantage. If Wikipedia has archival needs, they ask the Internet Archive to facilitate those needs.
(Plus they're expressly intended to last "forever" and are maintained by one of the few organizations with the multi-century history to credibly make that claim (Havard Law School))
https://perma.cc/about
But I would have used `.net` personally, rather than a country-specific domain.
You might have access through a registered institution (e.g., libraries and universities get it for free). Otherwise, the rates for individuals are hidden on their blog.[2]
If you really want to archive and shorten a link at once, https://archive.today (often seen on HN for bypassing paywalls) is another option, but its unknown ownership makes its long-term existence slightly questionable. For general web archival, the Internet Archive (https://web.archive.org) looks like it'll be around for a little longer. But since its crawler and server have their own quirks, I'd still submit links to multiple services for good measure.
[1] https://blogs.harvard.edu/perma/2019/01/07/introducing-indiv...
[2] https://blogs.harvard.edu/perma/2021/09/15/usage-plans-updat...
-----
Speaking of web archive reliability, it looks like WebCite might have recently died for good, after a decade of funding issues and intermittent downtime[3][4]—the first major web archive to fall?
[3] https://wiki.archiveteam.org/index.php/WebCite
[4] https://en.wikipedia.org/wiki/Talk%3AWebCite#2021-10-27
edit: https://github.com/harvard-lil/perma
EDIT: also for vanity "rememberable" urls
But for "vanity", the best vanity URL is a domain name. Buy it, own it.
foo.com/sites/blog/signup-for-my-conference-2022-seo-friendly-url
vs
shortenedurl.com/conf-2022
URL shorteners offering vanity urls solved people not needing to deal with the hassle of purchasing a vanity domain via a registrar, setting up DNS records, and waiting for propagation.
URL shorteners came later, and solved the problem of people needing to deal with “more steps” in order to have short memorable urls. Achieving memorable urls in “less steps” is one of the major selling points of url shorteners.
However, url shorteners did not solve short memorable urls — DNS had already solved that. But, as you’ve pointed out, acquiring a domain and setting up DNS records takes more work than simply using a url shortener service.
But really, opaque short URLs are significantly worse than the full URL: you have to go to extra effort to create them, and extra effort to retrieve from them, and you’re adding an additional point of failure, and you can no longer just read the URL.
People have many reasons they want a shorter URL. Twitter is only one of them.
Ok, maybe you mean mobile only: i have a wifi only android phone that I don't use email on, none of those options work.
There are worse things that could happen than shutting down too. Someone with control of an old shortener domain could set up spoof versions of the site that a link used to forward to and use it to harvest passwords or get people to install malware.
For example, if I go to some random product page on Amazon, the link looks something like this:
That link looks ridiculous and isn't suited for sharing in many situations. For most people, this is the link they really want: I know there are tools out there that can shorten URLs like that. I used to have a browser extension which did exactly that. It doesn't work for everything, but at least your link wont die because a third-party service shut down or messed up.Because of that it ultimately needs to a site-specific database/algorithm, perhaps with a fallback to the default behaviour like simply cleaning up the most common garbage like (_encoding/usg/etc). I suspect it's possible to use some sort of machine learning to guess the meaningful parts of the URL path/query/fragments, but even for that we need some human curation for the training set. I wish we could collaborate on a shared database/library for that, have sketched some ideas/applications/prior art here: https://beepb00p.xyz/exobrain/projects/cannon.html
I started thinking about it since I have a similar problem in Promnesia (https://github.com/karlicoss/promnesia#readme), a knowledge management tool I'm working on. Ideally I want to normalise URLS, so they address the exact bit of information, and nothing more.
Additionally, CleanURLs to the rescue! https://github.com/ClearURLs
And I know there are limitations to the concept. It wont work for pages which require authentication or any kind of session data. It wont work for pages which are intentionally dynamic. It wont work for sites which cannot be accessed by the service. It wont be useful for pages who simply have long URLs. But it probably covers the many common use cases for a URL shortener, and you could always fall back on traditional shortening methods (redirecting) when it doesn't work.
So it basically automates detecting useful bits for a particular URL, but it's kind of time consuming and flaky. It could be very helpful to populate the 'rules' database though, and then this database could be shared with other people so they don't have to scrape.
I guess when I said ML (or preferably some fuzzy algorithm/heuristic), I was referring to generifying rules so they also work on the sites not in the rules database. If humans can detect garbage in the URL looking at a few examples, the computer can too :)
Affiliate links used to catch some heat on HN but https://amzn.to seems to be showing up in more and more comments without too much blowback. It wasn't mentioned much that affiliates could view everything people bought while under their cookie, but I'm not sure if that's still possible.
No code version:
Maybe your internal systems are less prone to such complexity effects that turn "internal only" tools into "customer facing products" accidentally all the time, of course. Just that it's an interesting risk, well worth stating, and it is fun seeing all comments on shutdown notices of exactly one such "internal only" tool that accidentally became a "customer facing product" saying "just build it yourself". "Just build it yourself" doesn't entirely protect you from it being your "internal only" tool that's the next trending HN deprecation notice headline.
The "it's pretty trivial to do your own" piece is trivial if you already have gone through the effort of setting up an AWS account and finding a globally unique bucket name across all S3 users. That option by the way requires entering a CC, personal details, and requires you to pay expensive rates compared with shorteners which are typically free. Not to mention that the resultant URL is still comparatively long
Most people won't do that.
FWIW, most larger companies I've worked at have their own internal URL shortener that they run for all their internal docs and whatnot. That doesn't solve the public use case though.
Sure, most people won't do that. But the user base of github is not most people, it's developers, a large fraction of which work at a company which has an AWS account or equivalent, and I questioned specifically "companies" not "people" using an external service for this.
And the comments elsewhere on this thread should remind us all that "a lot can happen in forever"... My interpretation: eventually, everything will be deprecated. There may be some web sites (github.com, for example) that are "too big" (or "too important") to fail or disappear; but they eventually will.
[1] https://www.hyrumslaw.com/
I feel like there's a different "law" here, involving persistence and memory and history and recordability on the internet -- people want it, but it doesn't come for free, and there isn't necessarily anyone with a profit motive or business plan to make it happen.
I can never remember all the laws
I've been running https://T.LY URL Shortener for the past 2 years and I'm the creator of the URL Shortener Extension (https://t.ly/extension) that has over 400K users. Let me know if any questions around url shorteners.
Just yesterday I received a text from a shortcode with a "bby.me/XXXX" URL saying my Best Buy order was canceled. Turns out it was a legit order I placed nearly 6 months ago and was on backorder that I forgot about.
I'd love for a shortening service to offer an "unfurl" tool where I could paste in the URL, and it shows me what the redirected site is without clicking it and potentially activating their clickthrough tracking.
For Bitly, at least, you can add a plus sign to the end of a short URL to preview what the destination URL is. I don't know if other URL shorteners have something similar.
I played around with it and there was some seriously sketchy interstitial stuff going on.
No, wrong. Don't use a URL shortening service. Don't use any of them. Especially not in anything long-term.
Connecting a domain name to a piece of software running on any of your devices should require nothing more than a quick OAuth flow to your domain provider to connect a tunnel to the application. You shouldn't need to understand DNS, TLS, HTTP, ports, ip addresses, NAT, CGNAT, etc in order to self-host simple programs like URL shorteners quickly and securely.
Sometimes the content on the link doesn't exist anymore, sometimes the eg. newssite changed their url scheme, so the content still exists, but not on the same url, sometimes the media changed their story, so linking to the "same" article, now paints a different picture, and of course, sometimes old domains get hijacked by spammers... and more related, sometimes everything works except the url shortener.
Honestly, if I ever link anything that will be read after a long time, I only use archive.org/ links, and I cross my finger, that the snapshot will still exist in X years.
I don't understand this practice. It seems more proper to use the original URL and let readers themselves use archive.org when the link becomes dead.
Using the nonprofit's archives when you don't need them seems abusive. On average it should be simpler for the original server to render the page than for the wayback machine to fetch a particular snapshot. What's worse is that I don't think you're the only one needlessly using their archive like this, and wonder if, as a collective, uses like this significantly increase their operational costs.
Something happens, you link to cnn/fox/whatever, and discuss the news story or their comment or whatever, and two hours later, the whole article is rewritten, the headline is different and the subtone is different. This would be ok, if there was a "history" section at the beginning of the article, saying "this article was changed on date X and Y and Z, click here to see previous versions", but usually it's not.
an example widely publicized last year:
https://web.archive.org/web/20210525043631/https://www.nytim...
Then people get angry for newyork times using anti-semitic attacks for left-right politics, and soon after:
https://web.archive.org/web/20210525173242/https://www.nytim...
Different headline all together, same url.
Now if i said that ny times is using those attacs for politics, and linked the article directly, you (visiting the second version of the article) wouldn't find the headline "problematic", and you'd think i was just some far-whatever nutjob conspiracy theorist... by linking to the archive, you see the headline that I saw.
Git.io deprecation - https://news.ycombinator.com/item?id=31162829 - April 2022 (106 comments)