Because "URI" stands for "Uniform Resource Identifier" (or URL for "Locator"), not "Resource". The intent is for it to be a pointer, not the value at the location. And, if you were to use the URI as the content (instead of chunking and shrinking it via Bitly), you'd be duplicating that content on every page that links to it. And in your browsing history, by merely viewing it.
edit: oooh, another thought: you're essentially uploading the content of the page to view it.
There's an interesting copyright question in there somewhere too. If the URL for my document is the document then sharing the link is infringing my copyright, or something.
Is this really the case? Bit.ly would still have to forward the user over to the hashify.me site, where the hash would be decoded server-side and the content would have to be sent back over the wire to the client. That's still eating the same amount of bandwidth on hashify.me, no?
Bit.ly has to store the entire document encoded in base64 as the URL of the destination in their database in order to return to users the value of the given bit.ly URL hash. In essence, yes, bit.ly is storing the entire document on their servers anytime anyone shortens a hashify.me link.
Think if it like the difference between the postal service letting you know there is a package that you can go pick up at the post office, and the postal service giving you a package at your home or work that cannot be opened until you go to the store to buy a box cutter, but you have to bring the package with you.
The first example is cheap, since you only receive a pointer or link to where the package is, but you have to do all the work to get it. The second is not cheap, since if the package was a bed from Ikea (for a random large example), the postal service (bit.ly) has to deliver the package to you, and then you have to go somewhere (hashify.me) while carrying that package in order to see what's inside.
Ok, that makes sense. I thought we were debating on whether or not bit.ly incurred ALL the load and hashify.me incurred NONE, but that doesn't seem to be the case.
No. From what I understand, hashify does no intensive server side processing beyond providing the functions that allow the visitor to process the encoded string on its end; i.e., all intensive processing is done on the client's side.
But as people above mentioned, the data is still being uploaded unnecessarily, due to a little short sightedness by the hashify creator(s). The hash should be stored after a pound (#) character to ensure that the data isn't sent to the hashify server via the URL (i.e., the GET method), since all processing of the hash is done client-side anyway.
I can't recall the last thing that made me giggle as much as realizing what they were doing. I can't see any time I'd really use this, but the audacity is inspiring.
With a name like 'Hashify', I'm surprised they don't also offer the option of putting the content into the '#fragment' portion of the URL. Then, not even the hashify.net site would need to receive and decode the full URL; they'd just send down a small bit of constant Javascript that rebuilds the page from the #fragment.
The OP is saying that with a #fragment, the browser wouldn't even need to send the server the doc contents. (Since they aren't used anyway, there's obviously no use in doing so.)
I see this as a remarkable answer to the problem of needing to view a cached version of the website.
For example, what if a URL were posted to Hacker News, but after the URL was a ?hasifyme=THEHASH, where THEHASH was the Hash of the website linked-to.
This way, if the URL could not be loaded because the server load was to high, you could just forward the URL to Hashify.me and the cache of the plain text from the website would still then be readable.
Boom, instant cache of the website content stored right in the URL!!!!
That's not possible because the base 64 encoding is longer than the plaintext. There are 4 characters of text in the URL for every 3 characters of text entered.
If you're willing to jump through hoops and js-compress your content, you can deal with this issue, unless your compressor uses something like crc. That way, you can grow the plaintext faster than the b64 - though you'd still need to do clever things with the url.
bit.ly is the only URL shortening service to support cross-origin resource sharing, as far as I'm aware.
The _right_ thing to do would be to build a shortening service designed to handle URLs of arbitrary length. I'm not sure that I'm willing to take on that responsibility, though.
Actually tinyurl will shorten it but your browser may not allow the redirect. Chrome throws up an error page for security reasons: http://tinyurl.com/3maue6t
"Two-way hash function" is an oxymoron. The fundamental characteristic of a function that makes it a hash function is the inability to (feasibly) reverse it.
I agree with your final conclusion, that Base64 is an encoding, but it is also a hash function and hash functions can very well be two-way. In fact, the trivial hash function just maps data to itself, which is trivially reversible. However, we do want our cryptographic hash functions to be trapdoors to be of use.
Trapdoor function means the function is easy to calculate in one direction, but is difficult to calculate in the other direction without knowing a special bit of information that allows this calculation much easier.
What would a trapdoor value be for SHA1 ?
What I can agree is that I would like the cryptographic hash to be a one-way function, yes. But not trapdoor functions, please :-)
(and can you point to the definition of the hash functions that you described ? I'm curious).
> What would a trapdoor value be for SHA1 ?
Okay, you got me there. It's been long enough since I've actually used the term that I seemed to have remembered it being the same as one-way. Oops.
As for the parenthetical, I'm not sure I take your meaning properly. If you are asking where I learned that hash functions don't have to be one way it seems to be an odd question, but I just checked Wikipedia and it agrees with me, at least.
"Okay, you got me there." - it was not really intended as a trick - sorry for the wording.
As for the trivial hash - now I scrolled down the page on Wikipedia, indeed. I never thought of it this way. (That an identity function on an integer would deserve to be called a hash function :-)
The part that got me was the initial sentence about hash function converting "large, possibly variable amount of data" into a "small datum". As "large" and "small" are implied to be of different sizes, I glazed over a possibility of identity function there.
Oh, I didn't think you intended to trick me. Perhaps some context that'd help you understand my state of mind when I wrote that: while I may be a dev and hacker professionally and as a hobby, I went to university in mathematics specialising in number theory. I'm expected to know silly things like the difference between "one way function" and "trapdoor function", so it's a touch off-putting when I forget.
I know what you mean. For me it was: "Hm. he wrote that with a good confidence. Which part of my knowledge is wrong or incomplete ?" :-) Kind of the same feeling when the significant other garbage-collects a pen you put on the table just a few minutes ago.
Technically, if you squint hard enough, and assuming that Bit.ly always shortens the same URL to the same code (which on my brief testing it did when I tried the same URL from two different browsers), a one-way hashing function is being created from bit.ly shortened URLs -> the full "hashed" document. Technically. It certainly lacks most of the usual properties we associate with hash functions but the bare skeleton is sort of there. And there's no way to feasibly reverse that hash function algorithmically short of keeping the original input table and querying that, which is how Bit.ly of course works.
> Nice hack, though odd name given that no hashing occurs.
When I registered the domain I imagined that state would be stored in a Twitter-style hashbang. Thanks to `history.pushState` and `history.replaceState`, this hack is not required in modern browsers. :)
I hacked together an encrypted (aes 256) read/write "database" once with the bitty API as the persistence backend.
However, this site disappoints me, it doesn't seem to do anything other than what a data URL can do, except it's vulnerable to downtime because of a centralized website.
Edit: for those of you unfamiliar with what a data URL is. You an store a HTML or image document using a URL like data:text/HTML;base64,hashifystuffhere
This is going to break in cases where the request line grows above 8k-16k. Many browsers/proxies implement limits on headers/request lines, for good reasons.
This is quite clearly covered in the actual document!
> For longer documents, Hashify splits the contents into as many as 15 chunks. The chunks are then Base64-encoded and sent to bit.ly in a single request. The bit.ly hashes contained in the response are then "packed" into a URL such as http://hashify.me/unpack:gYi2Ie,g4fpte. Finally, this URL is itself shortened.
That's a feature. You can encode an entire webpage, including JavaScript, and, yes, including alerts.
As long as they don't have user accounts or database access or such, XSS doesn't let an attacker do anything meaningful. It's not weak security, it's just how the site works.
Agreed! One can always link someone to a static HTML page with <script>alert('fu')</script> in the body, but no one would tag that "XSS".
Does hashify.me make it easier to send annoying alert messages to your friends? Sure. Annoying, but no more harmful than sending them to the static equivalent.
Fair enough. It's similar to embedding third-party gadgets for things like iGoogle. However, in practice that content is sanitized.
Note there are risks to hosting arbitrary Javascript beyond stealing cookies. For example, you can steal browser history, discover NAT IP addresses, scan intranet ports, etc. Here's a presentation by Jeremiah Grossman covering some of these attacks:
http://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Gro...
Of course, attackers can host malicious content anywhere they control. I could just as easily send someone a bit.ly link to a malicious site I control.
Last week on a whim I whipped up a URL shortener that expires the forwarded URL after one week[1]. Using that plus hashify, you can essentially make expiring web pages.
I am just coding the same thing right now (began a week ago). Also had the idea to use bit.ly as shortener (because of its api) and make use of multiple shortened links to store the data. Right before looking at HN I was doing some research for a good js compressing algo.
On the one hand i am a bit disappointed (that i am too late), but on the other hand hashify.me is made far better I could make it. Great realisation.
It's interesting that you say that. For some reason I felt under immense pressure to get this out quickly, but I wrote that off as paranoia (should've heeded Kurt's warning).
Now that it _is_ out in the wild, I can't wait to see what people do with it slash build upon it.
Well, it really _was_ out before in many ways, but not in such a nice one (formated html etc.). Btw, do you have a base64 online converter and read my code six days ago? Just "kidding".
It is worth noting there's a difference in the base64.b64encode function, vs the default MIME Base64 codec on strings[2].
Per RFC2045[1]:
(Soft Line Breaks) The Quoted-Printable encoding
REQUIRES that encoded lines be no more than 76
characters long. If longer lines are to be encoded
with the Quoted-Printable encoding, "soft" line breaks
Due to the insertion of these soft-line breaks, encoding is not the same, as you can verify yourself:
import os
import base64
import unittest
class Base64Test(unittest.TestCase):
def test_long_string_base64_decoding_and_encoding(self):
byte_seq = os.urandom(500)
mime64_encoded = byte_seq.encode('base64')
self.assertNotEqual(base64.b64encode(byte_seq), mime64_encoded)
self.assertEqual(base64.b64decode(mime64_encoded),
mime64_encoded.decode('base64'))
if __name__ == '__main__':
unittest.main()
Decoding, as you can see above, is fine. This makes a difference when encoding a really long string in an HTTP header.
I could see this as a very useful implementation for HTML5/Mobile Web sites.
Consider the user experience for the target site on a mobile platform. You have already loaded the site on your mobile device before even taking action, so when you click the link the response is much faster than requesting the site at the click.
"A hash function is any well-defined procedure or mathematical function that converts a large, possibly variable-sized amount of data into a small datum"
No hashify is not really a hash function. Since a hash function takes large data to small data it is implicitly not invertible. Hashify's URLs obviously are.
And the relationship is this: 'path' element of bit.ly URLs are keys in a key->value mapping (where the value is your target URL), and one of the best implementations of a key->value mapping is a hash-table. (At least, it's good for in-memory implementations... I suppose that on disk something a little more elaborate may be called for?)
Historically, the authors of Perl and Ruby (and WP tells me, Common Lisp?) decided to confuse the interface with the implementation, and use "hash" or "hash table" to refer to the mapping, and not ever Perl hacker has a Computing Science degree, so now we live in a world of people who think that "hash" means the thing that bit.ly does for you.
In Common Lisp, it's not interface for mapping but really an hash table as some internal details of hash table implementation are exposed by the interface.
Clearly this is awesome, I'm curious as to what lead you to build it? Understanding that you weren't solving a 'problem', but you've created something really compelling here.
A month ago, all the developers in the office spent two days working on interesting small projects, the idea being that at the end of that time we'd have a bunch of cool shippable features. Though it was encouraged to work on useful, sensible things, this was not a requirement.
Anyway, I felt great until 4pm on the Friday, when we presented our creations. Afterwards, I felt flat (as one often does after meeting a deadline or finishing a series of exams). I didn't want the excitement to end.
As I was walking home I had an idea. For some reason I wanted to share thoughts in 72pt Helvetica. I didn't want to broadcast them (I was melancholic after all), but I felt compelled to express them visually.
I began to think about how this might be done. The Web seemed like the obvious platform. I wondered whether it could be done without a database. I remembered something I had heard on a podcast about a site that allowed one play musical notes on a computer keyboard, and would encode these in the URL for easy playback.
This seemed a lot more interesting than sharing my moody thoughts, and now that I had something cool to work on I no longer felt the need to do so anyway.
I think I spent 40 hours working on it that first weekend (yes, I was consumed). I truly believed that I could ship it before showering and leaving for work on Monday! Doing so would have been a mistake – I'm pleased that I spent several weeks ironing out the kinks and integrating with bit.ly and Twitter.
would be a good text "host" but needs clones, so that when it disappears in a few years I can still easily convert my urls back into the document therein. that's the one problem these text host sites have. they never last. this gets around this by hosting nothing, merely converting, but still.
and using bit.libya. i dont trust it.
isn't this also somewhat censorship resistant. since the hashify url without its bitly can be put anywhere on the web that is writable, thus making multiple copies available in a covert way.
This is pointless. It's impossible to create two pages that link to each other, for one. Also, as noted, most browsers won't allow URLs greater than 2k in size.
This research is a few years old but, hopefully, things should be even better by now: http://www.boutell.com/newfaq/misc/urllength.html .. Safari, Opera, and Firefox all go over 80k. I found another source for Chrome that says they "could not find any limits on Chrome and Safari".
Nonetheless, it's tricky because you have no idea if proxies in the middle will be able to cope, mobile clients, and all sorts of things.. so you're right in the sense that it's pointless (if you want it to be universally acceptable ;-)).
131 comments
[ 0.25 ms ] story [ 199 ms ] threadedit: oooh, another thought: you're essentially uploading the content of the page to view it.
The important thing is that the identified resource is unambiguously identified.
There might be legitimate uses for this, right now I can't think of one. Clever hack though.
Think if it like the difference between the postal service letting you know there is a package that you can go pick up at the post office, and the postal service giving you a package at your home or work that cannot be opened until you go to the store to buy a box cutter, but you have to bring the package with you.
The first example is cheap, since you only receive a pointer or link to where the package is, but you have to do all the work to get it. The second is not cheap, since if the package was a bed from Ikea (for a random large example), the postal service (bit.ly) has to deliver the package to you, and then you have to go somewhere (hashify.me) while carrying that package in order to see what's inside.
But as people above mentioned, the data is still being uploaded unnecessarily, due to a little short sightedness by the hashify creator(s). The hash should be stored after a pound (#) character to ensure that the data isn't sent to the hashify server via the URL (i.e., the GET method), since all processing of the hash is done client-side anyway.
Seriousy, though - awesome hack.
That's _exactly_ what the site does. Everything happens client-side. nginx serves a single index.html for every request. ;)
(You're uploading the data in order to then convert it client-side. Groxx noticed this too: http://news.ycombinator.com/item?id=2464347)
For example, what if a URL were posted to Hacker News, but after the URL was a ?hasifyme=THEHASH, where THEHASH was the Hash of the website linked-to.
This way, if the URL could not be loaded because the server load was to high, you could just forward the URL to Hashify.me and the cache of the plain text from the website would still then be readable.
Boom, instant cache of the website content stored right in the URL!!!!
I guess for this to work well. Dont you need Hashify to provider a service where you pass in hashcode and it returns the HTML to you.
You just wrap the call with good old JSONP and you are good to go.
Not so. hashify.me has no server-side component (beyond nginx serving three static files). _bit.ly_ kindly provides the hash table. ;)
Not having the time to do so, I leave the challenge here: http://bit.ly/ialoWI
The _right_ thing to do would be to build a shortening service designed to handle URLs of arbitrary length. I'm not sure that I'm willing to take on that responsibility, though.
data:text/html;charset=utf-8,However,%20data%20URI%20does%20the%20same%20without%20the%20server.
Here is a self-containing cached document, for simple texts probably with a better efficiency than base64.
EDIT: ah. The shorteners barf if you try to "shorten" the data URIs
With http://preview.tinyurl.com/3maue6t it works, btw.
Though, tinyurl still barfs when I try to shorten the data URI that http://software.hixie.ch/utilities/cgi/data/data gave to me when I let it grab the index of HN.
(This has always been an idea I was thinking about; nice to see it implemented)
That way you only get the text of the page, stored in a cache.
ERROR
The requested URL could not be retrieved
While trying to process the request:
GET ignifero ↗ For some reason i always thought URLs have a max length of 1024. Is there anything about it in the HTTP standard? HelloBeautiful ↗ There is no max length in the standard. Of course every browser has a max it can handle. For IE it is 2000 characters. mikecane ↗ Doesn't work in latest version of Opera: Connection closed by remote server
Check that the address is spelled correctly, or try searching for the site.
Base64 is an encoding.
What would a trapdoor value be for SHA1 ?
What I can agree is that I would like the cryptographic hash to be a one-way function, yes. But not trapdoor functions, please :-)
(and can you point to the definition of the hash functions that you described ? I'm curious).
As for the parenthetical, I'm not sure I take your meaning properly. If you are asking where I learned that hash functions don't have to be one way it seems to be an odd question, but I just checked Wikipedia and it agrees with me, at least.
As for the trivial hash - now I scrolled down the page on Wikipedia, indeed. I never thought of it this way. (That an identity function on an integer would deserve to be called a hash function :-)
The part that got me was the initial sentence about hash function converting "large, possibly variable amount of data" into a "small datum". As "large" and "small" are implied to be of different sizes, I glazed over a possibility of identity function there.
Thanks!
When I registered the domain I imagined that state would be stored in a Twitter-style hashbang. Thanks to `history.pushState` and `history.replaceState`, this hack is not required in modern browsers. :)
Here's what I'm seeing in my browser console:
{ "data": [ ], "status_code": 403, "status_txt": "RATE_LIMIT_EXCEEDED" }
I should have included appropriate error handling for this! Everything except shortening continues to function, though.
However, this site disappoints me, it doesn't seem to do anything other than what a data URL can do, except it's vulnerable to downtime because of a centralized website.
Edit: for those of you unfamiliar with what a data URL is. You an store a HTML or image document using a URL like data:text/HTML;base64,hashifystuffhere
Just sayin' ;)
Or just, you know, email people the content if you can give them the same data anyway.
It's a very cool idea though.
> For longer documents, Hashify splits the contents into as many as 15 chunks. The chunks are then Base64-encoded and sent to bit.ly in a single request. The bit.ly hashes contained in the response are then "packed" into a URL such as http://hashify.me/unpack:gYi2Ie,g4fpte. Finally, this URL is itself shortened.
copy that to hashify, and let me know the length of the url you get.
Yes, 15 * 2048 = a 30k limit on document size
As long as they don't have user accounts or database access or such, XSS doesn't let an attacker do anything meaningful. It's not weak security, it's just how the site works.
Edit: To point out the obvious, your iframe trickery is not necessary. Script tags are not escaped, nor are event attributes: http://hashify.me/PGRpdiBvbmNsaWNrPSJhbGVydCgnVGhpcyBpc25cJ3...
Agreed! One can always link someone to a static HTML page with <script>alert('fu')</script> in the body, but no one would tag that "XSS".
Does hashify.me make it easier to send annoying alert messages to your friends? Sure. Annoying, but no more harmful than sending them to the static equivalent.
Note there are risks to hosting arbitrary Javascript beyond stealing cookies. For example, you can steal browser history, discover NAT IP addresses, scan intranet ports, etc. Here's a presentation by Jeremiah Grossman covering some of these attacks: http://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Gro...
Of course, attackers can host malicious content anywhere they control. I could just as easily send someone a bit.ly link to a malicious site I control.
[1] pygm.us
On the one hand i am a bit disappointed (that i am too late), but on the other hand hashify.me is made far better I could make it. Great realisation.
Now that it _is_ out in the wild, I can't wait to see what people do with it slash build upon it.
Instead of...
You can do... Encoding works too. As well as zip (foo.encode('zip')).Per RFC2045[1]:
Due to the insertion of these soft-line breaks, encoding is not the same, as you can verify yourself: Decoding, as you can see above, is fine. This makes a difference when encoding a really long string in an HTTP header.[1] http://www.ietf.org/rfc/rfc2045.txt [2] http://docs.python.org/library/codecs.html#standard-encoding...
Consider the user experience for the target site on a mobile platform. You have already loaded the site on your mobile device before even taking action, so when you click the link the response is much faster than requesting the site at the click.
Hashify is not really a hash, is it?
The difference is that AFAIK there's no algorithm to take a URL (plus or minus a username) and give you a bit.ly ID, short of looking it up at bit.ly.
Historically, the authors of Perl and Ruby (and WP tells me, Common Lisp?) decided to confuse the interface with the implementation, and use "hash" or "hash table" to refer to the mapping, and not ever Perl hacker has a Computing Science degree, so now we live in a world of people who think that "hash" means the thing that bit.ly does for you.
Once again, Larry Wall Ruins Everything. :P
I love this little hack. Sure, it may have no practical purpose; but it gave me great joy to see this. I'm still smiling.
Care to give a peak into how you came up with it?
Anyway, I felt great until 4pm on the Friday, when we presented our creations. Afterwards, I felt flat (as one often does after meeting a deadline or finishing a series of exams). I didn't want the excitement to end.
As I was walking home I had an idea. For some reason I wanted to share thoughts in 72pt Helvetica. I didn't want to broadcast them (I was melancholic after all), but I felt compelled to express them visually.
I began to think about how this might be done. The Web seemed like the obvious platform. I wondered whether it could be done without a database. I remembered something I had heard on a podcast about a site that allowed one play musical notes on a computer keyboard, and would encode these in the URL for easy playback.
This seemed a lot more interesting than sharing my moody thoughts, and now that I had something cool to work on I no longer felt the need to do so anyway.
I think I spent 40 hours working on it that first weekend (yes, I was consumed). I truly believed that I could ship it before showering and leaving for work on Monday! Doing so would have been a mistake – I'm pleased that I spent several weeks ironing out the kinks and integrating with bit.ly and Twitter.
and using bit.libya. i dont trust it.
isn't this also somewhat censorship resistant. since the hashify url without its bitly can be put anywhere on the web that is writable, thus making multiple copies available in a covert way.
Nonetheless, it's tricky because you have no idea if proxies in the middle will be able to cope, mobile clients, and all sorts of things.. so you're right in the sense that it's pointless (if you want it to be universally acceptable ;-)).
I found that my web server started throwing 413s after only about ~30k characters.