I was always fixated with URL shorteners and always wanted to write one myself. While it would be easy to write one that works with a server, I loved the idea of hosting and running one for free.
Thus, I wrote this URL shortener which does not need a backend to work at all. It uses GitHub issues as a "database", and unlike most other URL shorteners that do not need backends, this one doesn't need a # prefixed to the alias. (i.e. short URLs look clean like this: nlsn.cf/1 instead of looking like this: nlsn.cf/#1)
I haven't had the time to work on it more but do let me know what you guys think! Thank you and hope yall enjoy it!
Looks nice and simple! Bit strange to claim "there is no database" and then add "GitHub issues is the database", maybe add "there is no traditional RDMS/document database needed" or something similar.
You might want to detect infinite loops as well :) https://nlsn.cf/6
Last note, the status code seems to be incorrectly set to 404 instead of any of the redirection status code. If you're hosting it on GitHub pages, I'm not sure you can actually control it. If the status code was correctly set + the Location header, I think all browsers would be able to detect the infinite loop automatically and prevent it.
If you use it personally and you know what you are doing, I think this idea is good. For example I have my own domain and when I share long link to friends, I can use it.
By the way, your feedback is good for author improves. Thank you.
HAHA so that was you. That was a good one! Yes and that title is definitely slightly misleading I kinda didn't want it to get so long and wanted that slight clickbait effect.
Yep I actually cannot control the status code at all. Part of what made this a fun and hacky project was the fact that I was actually redirecting to the actual URLs by catching the 404s. So, the only way i could think of catching the infinite loop was to supply the domain. That being said, I do appreciate the feedback and it's always great to learn more!
I think you could catch the infinitive loop client-side as well by comparing what the URL to redirect to is, and comparing it with the current URL. If they are the same, prevent the redirect. Although you would still be able to create loops by using two issues instead. Maybe just comparing the full hostname? If it's nlsn.cf, don't redirect.
"I was always fixated with URL shorteners and always wanted to write one myself. While it would be easy to write one that works with a server, I loved the idea of hosting and running one for free."
I felt exactly the same way and did, in fact, build one.[1]
It was pretty simple. There's an interesting twist to the particular one I built - it's not specifically a URL shortener, it's a "Universal Shortener".[2] Also no tracking or cookies or third party connections, which I find pleasing.
The disappointing part is seeing the malicious spam and phishing URLs that get shortened. We have had to manually flag-hold-approve a lot of strings and IIRC we outright block anything with 'paypal' in it.
We also gave a kill-function to our upstream ISP so that they could immediately act on phishing complaints without our having to be involved in real-time.
Oh that is cool! It does remind me of https://telegra.ph/ by Telegram! (Even though they weren't really advertising it as a universal shortener). Just a quick question, how did you promote your platform!
I've spent almost zero time on "Oh By" in the last 24 months as things at rsync.net have been very busy. So, we haven't made a sale in months due to lack of publicizing it.
However, I think there are unknown, emergent use-cases out there for "Oh By" that I hope people will discover. I am going to pivot back to "Oh By" next year ...
My biggest problem with this is the domain name. If you say "oh by", a vanishingly small amount of people will translate this to "0x". And you can't say "oh ex dot co", because the URL has both a 0 and a O in it - which will also always be ambiguous when written down. So you have to say something like "zero ex dot co", which I suppose is fine, but it has a lot of extra syllables.
This is brilliant! Especially to distribute branded shortened URLs of the kind {mydomain}/{id} without having to maintain pretty much anything and giving the end user a chance to verify what the URL actually leads to. Thank you!
I would hope that the most they’d do is limit/throttle the number of new and updated issues. As long as the limit is very generous for a normal project, this sort of thing would still be supported at reasonable usage levels.
Hopefully it wouldn’t be considered abuse just by its nature (as opposed to any traffic spikes it might cause). Quite the reverse, this shows off the flexibility of the service!
This is definitely just a fun hack and not meant for production! For production ready short links with low overheads I would personally recommend using Firebase URLs or cloud firestore.
I reckon it'll be possible to "simplify" it even more for an end-user, by making a static site generator generate these redirect HTML files from markdown or a CSV.
Some guy shared his project in this post that does exactly that! Yes while that is possible and can get to a high level of automation via GitHub actions, the point of my URL shortener was more of people could post a link easily with a simple interface (like the interface for adding a new issue). But if I were operating with a fixed set of short links, then this would have been a good method!
Indeed, especially with the recursive link created by someone else (nlsn.cf/6)
GitHub does have a 60 calls per hour API limit so I don't think it affects them as much(?) That being said this project is meant to be for fun and definitely not recommended for production purposes HAHA
Think of this as the software version of graffiti.
It is an artistic and creative way to shorten urls, but likely won’t exist forever. As long as it does though, we can appreciate the art for what it is.
The best hack on github issues I know is the trending-repos repo. Just subscribe to the issue of the technologies you care and you'll receive a daily/weekly email with the most starred repos for that period:
I've seen quite a few uses of the issues page as a discussion forum too. I certainly wouldn't consider any of it "abuse", however.
I wonder if anyone has made this observation before in regards to such things: "Anything that can be used to store arbitrary data, will be used to store arbitrary data."
I always wondered if relying so much on external libraries in the long run is more hassle to maintain than doing it by yourself. Security updates will be provided for X major version for a reasonable amount of time, but afterwards you will have to update, check that nothing breaks and update your code as well. Also some of the deps will break regardless of SemVer and you will need to update.
My fault. Didn't know that ES2015 actually supported `.then()`. Anyway, this was also an exercise for me to learn about babel (I use CRA all the time so have no idea what is going on behind the scenes) and this was a nice intro!
I figured that since the dependencies were only used in the build step it's not really considered. That being said, I didn't know ES2015 had `.then()` and was kinda trying out babel since I always used CRA and never knew what was going on behind the scenes. The dependencies were definitely not needed but in the production build, they aren't really "unminimal"
Essentially this is the promise of the blockchain. Imagine one massive global database where anyone can store anything in encrypted form. Free, forever.
Paying puts your transaction near the head of the queue. This matters more when there's a backlog. If you're willing to wait for things to clear up, you don't need to pay. https://bitcoinfees.net/
I've seen people suggest using pi as a "database." In this instance you could map ranges of pi to characters. Then you search pi until you find the right range. The URL would contain the range(s) needed.
This is terrible and so computationally wasteful. But it's interesting to think about.
In a similar vein, almost every algorithm can be converted into a O(1) space version simply by converting it into the dumbest possible search algorithm over the solution space: just sample from the solution space randomly (with a seeded PRNG or something) until you get the right answer.
Interestingly, bogosort cannot actually be implemented in constant space -- at least, not if you're talking about a deterministic algorithm. In order for your PRNG to have a large enough state space to cover all possible permutations, it needs O(n log n) bits.
In the Turing machine model of computation, algorithms that use O(1) space are theoretically equivalent to finite state machines (potentially with a very large number of states). Logarithmic space is where things start to get interesting, because it's enough to store a constant number of pointers into an arbitrarily large input.
Seems technically possible - example.com/12_3456 would calculate out pi, read 12 digits starting at position 3456, then convert those digits to ascii and you'd have your result. Given how far into the digits pi you'd have to go to find the exact right sequence that matches, eg, https://google.com, though, I suspect the resulting position number would be longer than the original url!
Also, conceptually, it seems like you're not so much using pi as a database as you are using it as an encoding mechanism: turning "abc.com" into a string of numbers whose length is relative to the original string. At that point you might as well pick an easier encoding, like base64. :)
You could add a short one or two character long identifier that changed the digits-of-pi to ascii map then select the mapping that results in the lowest index. In this case part of the "database" would exist within the URL shortener itself.
"X in pi" is an interesting thought experiment, but practically speaking, the least efficient way to do anything, since finding a string of N digits in pi will require a search space exponentially much larger than N on average, so it ends up just being an extremely inefficient encoding. A huffman coding + base32/base64 is a better "databaseless" short URL system.
Haha this is fantastic. Are the IDs still coming from that single counter lambda? Or put another way, do you now have 1,750,785 lambdas in your AWS account? :)
Yeeeeeeep. :) Someone automated spinning up new IDs at some point and flooded this service with urls. I wrote a script to delete old ones, but it takes a few days to run and I haven't gotten around to letting it finish.
But this has a chance of not working since you are doing the redirection client-side. Some addons block 3rd party resources. Looks like you're using the github API and sending the request from the user's browser.
Since I am running a URL shortener service https://blanq.io, my two cents:
1. I looked at https://github.com/nelsontky/gh-pages-url-shortener/blob/mai... and it is a basic script that does nothing more than redirection. A lot of URL shorteners on the web do a lot more. Premium ones can track users and route links based on client.
2. Most of URL shortening audience is non-technical and they usually like to pay and forget about it. Hosting and running your own service is an overhead.
3. Handling traffic at scale is challenging. So, if you have a lot of hits, I would not advise spending time maintaining the service.
How come I couldn’t find your service when I was looking for alternatives for bitly? The only results I got via Google, product hunt ... services that were overpriced and very limiting (nr of teammates)
Yes I agree with all of the above. And with my fixation on URL shorteners, I did have a dream of creating a full fledged URL shortening service like yours. I wasn't technically skilled enough for that then and thus didn't do it but your site does look great!
Thanks. I started this since it had a low-entry barrier but, going from basic to a mature product was quite challenging and a great learning experience personally.
That is quite interesting. Only issue is to rely on Github Issues for it (it might get messy with too many issues opened).
I had this same ~dream of hosting a URL Shortener statically, which lead me to create URLZap. The difference is that it relies on a config file and it can be used on any code repository service https://github.com/brunoluiz/urlzap
Cool project! Thanks for sharing! Yea the reason why I chose GitHub issues as a "database" was because it is simple to add a short link that manner. Definitely may get messy and there are other shortcomings like the lack of ability to indicate/reuse aliases. The other way to run a URL shortener service with low effort would be to use Firebase. Your idea is a cool one, love it especially since it uses GitHub actions too!
There's a lot of useful stuff to learn from this tiny example, providing you look at it more as a learning exercise than a production enterprise solution!
The github api does not care about your origin so that wasn't something I had to deal with! If I had to deal with something like that, my best option would be to use a proxy server, which would arguably negate the whole point of this "serverless" hack.
Your browser should, though. I would expect the fetch() request from nlsn.cf to github.com to throw a security error — presumably, the way custom domains are handled on github pages includes the addition of the custom domain as a trusted source at the github end.
What I meant was that GitHub's API does not check for the Host header, and the API allows connections from any source. Thus, CORS isn't an issue at all.
No, when a script tells a browsers to make a cross-origin request such as a GET or POST, the browser first makes a "pre-flight" request (without the payload) using the OPTIONS method to see what CORS-related response headers come back. If headers are returned that allow for it to proceed, the browser then makes whatever request the script asked for.
The network tab of developer tools should reveal all of this.
It should, in the right circumstances. By default, CORS dictates that cross-origin requests should not be allowed.
But sometimes you want that to be possible, so we have headers available to is where we can signal which domains are allowed to access the origin when on another origin.
In the case of the GitHub API, they (GitHub) are setting these headers to allow any origin to access the GitHub API from any other origin, that's why your browser doesn't throw a security error. Check out the various "access-control-*" headers the GitHub API returns as response headers when you use it.
Ah, thanks. I thought they might've had a dynamic list of domains that they add each custom domain too; I guess that would be enormous, even if it sounds a lot more secure than "Access-Control-Allow-Origin: *"!
I know perfectly fine what you mean here, but in the name of security, it's important to be precise.
All websites and browsers have CORS, one way or another, as CORS is the general concept. By default, only "same-origin" requests are allowed and "cross-origin" requests are disabled. But CORS is still there none the less.
What GitHub has done in this case, is add support for "cross-origin" requests.
Nitpicky maybe, but thought it'd be useful to add to avoid any confusion.
Such a fun little hack. The killer feature is that you found a way for anyone to create a new short URL without a pull request or any changes to the original repository. Nice work.
Neat project. What were your ideas for storing the content in git version history? (i.e as files itself?)
Shameless plug for a similar project that i did few years back: A (partial) static site generator based on github issues :)
https://github.com/geekodour/gitpushblog
Congrats! I really love this concept for two reasons.
First I'm always fascinated by smart solutions that make use of existing free (as in beer) infrastructures to provide a service with open-source software. In this way we truly get FOSS.
Second, I really like the idea that the information on where the shortened url redirects to is publicly available. I know that solutions like bitly.com do provide a way to preview the shortend url, but I think this is just more transparent, although admittedly a little bit less obvious.
You raise a point that strikes me as odd, but perhaps I am not up to date on what is considered FOSS nowadays.
In what way will this lead to `truly` FOSS? Sure, the URL shortener project is open-source, but neither Github Pages nor Github Issues are free-as-in-libre open-source projects. What is being show-cased here is a neat idea, but ultimately the dependency on Github's Pages and Issues will make it difficult to port to other systems, thus you are not really free to do with as you please.
So, while the URL shortener program itself is FOSS with all the right licenses, doesn't the direction you are proposing lead to more capture by these non-free platforms, as the value they offer for free-as-in-beer becomes more and more difficult to ignore?
Better be ready for the malicious links. I'm the developer of https://t.ly/ link shortener. I spend most of my time adding in protection to prevent people from using the service for malicious links.
Thank you for the advice! I love that domain though, have always been trying to find a 4 character URL shortener since GoDaddy killed x.co. Will definitely use your service more than I use my own HAHA. That being said, my project is not meant to be seriously used, just kinda a cool hack that is not meant to work forever/work reliably. Appreciate your comments and all the best for your service! You just gained one more (non) malicious user!
Thanks! Your project is really neat. Curious how long github would let it go and if they have a max number of issues. In a short time, T.LY has almost 5 millions short links. I have a lot of users using the shortener from the extension https://t.ly/extension
181 comments
[ 0.30 ms ] story [ 207 ms ] threadThus, I wrote this URL shortener which does not need a backend to work at all. It uses GitHub issues as a "database", and unlike most other URL shorteners that do not need backends, this one doesn't need a # prefixed to the alias. (i.e. short URLs look clean like this: nlsn.cf/1 instead of looking like this: nlsn.cf/#1)
I haven't had the time to work on it more but do let me know what you guys think! Thank you and hope yall enjoy it!
You might want to detect infinite loops as well :) https://nlsn.cf/6
Last note, the status code seems to be incorrectly set to 404 instead of any of the redirection status code. If you're hosting it on GitHub pages, I'm not sure you can actually control it. If the status code was correctly set + the Location header, I think all browsers would be able to detect the infinite loop automatically and prevent it.
By the way, your feedback is good for author improves. Thank you.
Yep I actually cannot control the status code at all. Part of what made this a fun and hacky project was the fact that I was actually redirecting to the actual URLs by catching the 404s. So, the only way i could think of catching the infinite loop was to supply the domain. That being said, I do appreciate the feedback and it's always great to learn more!
I think you could catch the infinitive loop client-side as well by comparing what the URL to redirect to is, and comparing it with the current URL. If they are the same, prevent the redirect. Although you would still be able to create loops by using two issues instead. Maybe just comparing the full hostname? If it's nlsn.cf, don't redirect.
Doesn't work; eg, make http://someothershorten.er/foo -> http://nlsn.cf/1234 -> http://someothershorten.er/foo.
https://news.ycombinator.com/item?id=25113154
GitHub issues infrastructure is a backend!
I felt exactly the same way and did, in fact, build one.[1]
It was pretty simple. There's an interesting twist to the particular one I built - it's not specifically a URL shortener, it's a "Universal Shortener".[2] Also no tracking or cookies or third party connections, which I find pleasing.
The disappointing part is seeing the malicious spam and phishing URLs that get shortened. We have had to manually flag-hold-approve a lot of strings and IIRC we outright block anything with 'paypal' in it.
We also gave a kill-function to our upstream ISP so that they could immediately act on phishing complaints without our having to be involved in real-time.
[1] https://0x.co
[2] https://0x.co/hnfaq.html
I also sent a notice to all rsync.net customers, years ago, when we launched it.
That's it.
I've spent almost zero time on "Oh By" in the last 24 months as things at rsync.net have been very busy. So, we haven't made a sale in months due to lack of publicizing it.
However, I think there are unknown, emergent use-cases out there for "Oh By" that I hope people will discover. I am going to pivot back to "Oh By" next year ...
Then google can just find that index page and index every link on it - and all codes marked searchable will be indexed by google.
What if some of shortener domain so popular that Github takes some action? Like block or ban?
Hopefully it wouldn’t be considered abuse just by its nature (as opposed to any traffic spikes it might cause). Quite the reverse, this shows off the flexibility of the service!
<meta http-equiv="refresh" content="1; url='https://news.ycombinator.com/'" />
in the header.
https://gitlab.com/davch/static-redirect
GitHub does have a 60 calls per hour API limit so I don't think it affects them as much(?) That being said this project is meant to be for fun and definitely not recommended for production purposes HAHA
It is an artistic and creative way to shorten urls, but likely won’t exist forever. As long as it does though, we can appreciate the art for what it is.
https://github.com/vitalets/github-trending-repos/
I wonder if anyone has made this observation before in regards to such things: "Anything that can be used to store arbitrary data, will be used to store arbitrary data."
Like licensing arguments?
https://github.com/nelsontky/gh-pages-url-shortener/blob/mai...
it does need a database though. it just uses someone else's database
This is terrible and so computationally wasteful. But it's interesting to think about.
A well-known example of this is bogosort: https://en.wikipedia.org/wiki/Bogosort
In the Turing machine model of computation, algorithms that use O(1) space are theoretically equivalent to finite state machines (potentially with a very large number of states). Logarithmic space is where things start to get interesting, because it's enough to store a constant number of pointers into an arbitrarily large input.
Also, conceptually, it seems like you're not so much using pi as a database as you are using it as an encoding mechanism: turning "abc.com" into a string of numbers whose length is relative to the original string. At that point you might as well pick an easier encoding, like base64. :)
But this has a chance of not working since you are doing the redirection client-side. Some addons block 3rd party resources. Looks like you're using the github API and sending the request from the user's browser.
1. I looked at https://github.com/nelsontky/gh-pages-url-shortener/blob/mai... and it is a basic script that does nothing more than redirection. A lot of URL shorteners on the web do a lot more. Premium ones can track users and route links based on client.
2. Most of URL shortening audience is non-technical and they usually like to pay and forget about it. Hosting and running your own service is an overhead.
3. Handling traffic at scale is challenging. So, if you have a lot of hits, I would not advise spending time maintaining the service.
4. Github can kill this anytime.
That said, this is neat solution for basic needs.
And target enterprises with custom URL's. They look very good
Maybe copy some features adjust has (deep linking and redirecting to respective app stores with tracking parameters)
https://www.adjust.com/glossary/deep-linking/
Businesses that I target have set up custom domains and reap more benefit from it.
I eventually plan to retire default shortening domain(blanq.io) and only offer custom domains.
2. who "likes to pay"? Having a free option is great, no matter whether you're "non-technical" or not...
3. It's hosted on GH Pages. I think they're pretty okay on the scaling front, what do you think?
4. Technically yes, as in "they could switch off GH Pages and the API". Not sure how likely that is. Also, the redirections are saved as GH issues
2. Businesses like to pay.
3. Github has been bad lately https://www.theregister.com/2020/07/13/github_takes_some_dow...
https://en.wikipedia.org/wiki/.re
Examples are youtu.be , goo.gl , redd.it , cbsn.ws
I had this same ~dream of hosting a URL Shortener statically, which lead me to create URLZap. The difference is that it relies on a config file and it can be used on any code repository service https://github.com/brunoluiz/urlzap
OP, how does this get around CORS?
The network tab of developer tools should reveal all of this.
It should, in the right circumstances. By default, CORS dictates that cross-origin requests should not be allowed.
But sometimes you want that to be possible, so we have headers available to is where we can signal which domains are allowed to access the origin when on another origin.
In the case of the GitHub API, they (GitHub) are setting these headers to allow any origin to access the GitHub API from any other origin, that's why your browser doesn't throw a security error. Check out the various "access-control-*" headers the GitHub API returns as response headers when you use it.
It has nothing to do with custom domains or GitHub Pages, but all to do with CORS and associated headers. You can learn more about CORS here: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
I know perfectly fine what you mean here, but in the name of security, it's important to be precise.
All websites and browsers have CORS, one way or another, as CORS is the general concept. By default, only "same-origin" requests are allowed and "cross-origin" requests are disabled. But CORS is still there none the less.
What GitHub has done in this case, is add support for "cross-origin" requests.
Nitpicky maybe, but thought it'd be useful to add to avoid any confusion.
Access-Control-Allow-Origin: star
Shameless plug for a similar project that i did few years back: A (partial) static site generator based on github issues :) https://github.com/geekodour/gitpushblog
First I'm always fascinated by smart solutions that make use of existing free (as in beer) infrastructures to provide a service with open-source software. In this way we truly get FOSS.
Second, I really like the idea that the information on where the shortened url redirects to is publicly available. I know that solutions like bitly.com do provide a way to preview the shortend url, but I think this is just more transparent, although admittedly a little bit less obvious.
In what way will this lead to `truly` FOSS? Sure, the URL shortener project is open-source, but neither Github Pages nor Github Issues are free-as-in-libre open-source projects. What is being show-cased here is a neat idea, but ultimately the dependency on Github's Pages and Issues will make it difficult to port to other systems, thus you are not really free to do with as you please.
So, while the URL shortener program itself is FOSS with all the right licenses, doesn't the direction you are proposing lead to more capture by these non-free platforms, as the value they offer for free-as-in-beer becomes more and more difficult to ignore?