the URL creation and the browsing of a short URL definitely works fast, I guess this being on the front page is a cheap way to verify if the C10k claims are true!
CloudFlare is a perfect companion for something like this because you can hard cache the redirects on their edges. I have a few services that run off tiny boxes, and just leverage CloudFlare free edge caching.
Perhaps I am the only one, but is anyone interested in the opposite direction, static site generator like CLI app to do short linking?
YOURLS was popular for a while, and I tried it, but I was concerned with running a not very popular PHP app even on shared hosting. At least Wordpress gets decent attention. I was worried of people compromising my own YOURLS instance against me.
No idea about equivalents for nginx, or something that could run on Github Pages (these would probably be the same, given that nginx doesn't have an equivalent to .htaccess out of the box).
I am going to check out shurl, very interesting. It does sound familiar but I cannot remember why. Sometimes I see these things and forget to bookmark them later.
Looking at the coding patterns used in the C source, I am utterly horrified this is running live on a public facing website.
I can see at least one buffer overrun dependent on database contents, and I wouldn't be surprised if there's public-facing vulnerabilities in this thing, but I don't want to spend another 5 minutes looking.
That still counts as a vulnerability, even if it's not exploitable without also being able to write the correct state into the database. Databases almost always contain user-generated data. I don't trust the data contained therein any more than I trust the user. Validate it every time.
That sounds a bit too much. The data from db is supposed to be already validated.
Do you validate the input received from the validation function as well?(⸮)
I'm not saying you need to fully validate the database, but at least making sure the row you're about to read can actually fit in the buffer you just allocated would be a good practice, even if the input validation shouldn't normally allow it.
If the database could theoretically hold it according to its schema, the program should be prepared for the largest row size that is still technically legal. (Where "prepared" may simply mean that it throws a "This should never happen" error and aborts the request.)
I don't think the solution is to validate it everytime you use that data/value. I fail to see how you mitigate the issue with the second validation. You have duplicate records so hoe you fix it? Shouldn't you check duplicate sccounts on sign up?
You're still thinking too small. It isn't about validation for duplicate accounts, and by the way, this gets past that duplicate account validation, because "admin" is different from "admin' --"
You get around it by parameterizing your queries, and not blindly trusting data that comes from the database.
I know, this is some internal glibc allocation code failing because that request overflowed a buffer on the heap. The bug is, as tptacek mentioned, in the header receiving function [1].
The compilation warnings was an additional remark, I should've phrased that post better.
Nothing is wrong with malloc per se, but the way it's used here is extraordinarily unsafe (they're allocating fixed-sized buffers for variable-length fields). Even if they'd attempted to allocate variable-length buffers, they'd still need to handle integers safely, and this code doesn't.
There are other needs for a shortened URL. Perhaps you have to hardcode a link in code and don't want to be tied to a page whose contents may break. If you control your URL shortener, you can hardcode your shortened one, and update the redirect as needed.
> Perhaps you have to hardcode a link in code and don't want to be tied to a page whose contents may break. If you control your URL shortener, you can hardcode your shortened one, and update the redirect as needed.
And when your URL shortener goes down, all links are dead. Shortening URLs is the last thing you want to do when you need to preserve pages that might go down. Caching is a better solution.
We[1] just launched a new shortener and it supports URLs.
That's not the primary use-case for it[2][3], but we're happy to support people shortening URLs if they want to. The reason there's room for this is that our URL shortening function has no ads, no third party tracking, no bloat ... no dark patterns.
> The reason there's room for this is that our URL shortening function has no ads, no third party tracking, no bloat ... no dark patterns.
But it’s still a closed, proprietary database and there’s no way to decode an “Oh By Code” if the service goes down, which is the disadvantage number one you have with URL shorteners.
> The real utility are the easily recognizable codes, prefixed with "0x".
Why not using a prefix that helps differentiate your codes from hexadecimal numbers? How do you plan to get people to know your service if it’s not recognizable? Let’s say I put one of those codes on my business card, I still have to write somewhere that people should use 0x.co to access the content, which ruins the advantage of having just one code rather than a URL to my website (literally anyone knows how to open a URL in a browser).
"But it’s still a closed, proprietary database and there’s no way to decode an “Oh By Code” if the service goes down, which is the disadvantage number one you have with URL shorteners."
Yes, that's correct. In this case, "Oh By" will continue running indefinitely. This is true for two reasons:
1) Building an extremely lightweight, ad free interface has a happy byproduct of ... being extremely lightweight. The infrastructure requirements for even a wildly popular "Oh By" are trivial.
2) Because Oh By is self-funded, there is no pressure from outside parties to break the pattern of an ad free, tracking free service with zero bloat.
If you're having trouble swallowing this, remember that rsync.net has been running continuously since 2001[1]. I think that's a credible track record[2].
[1] As a feature of JohnCompanies, the first VPS provider, and then as a standalone corporation in 2006.
[2] In fact, the rsync.net warrant canary turns 10 years old this year.
"How do you plan to get people to know your service if it’s not recognizable? Let’s say I put one of those codes on my business card, I still have to write somewhere that people should use 0x.co to access the content, which ruins the advantage of having just one code rather than a URL to my website (literally anyone knows how to open a URL in a browser)."
Well that's the real trick, isn't it ?
I'm happy to report that while Oh By itself does not display ads, we are a consumer of ads. We have a decent budget for sales and marketing.
I like to use single line URLs in a number of circumstances, particularly email and IRC. While some URLs can be trimmed (excessively long Amazon or eBay URLs for instance), others just need a URL shortener, and I'd prefer to use one I control.
I used to run a public one but it was getting abused for spam so I stopped.
(edit: and you can imagine how useful that's going to be once browsers start checking the Wayback whenever they see a 404... we're getting close to starting a trial of that with Firefox, with other browsers to follow.)
Is anyone suffering from lack of URL shorteners? Or does anyone really care what their URL shortener is implemented in? Is there even any way that the application code contributes to the request time more meaningfully than the persistence mechanism used?
Cool hacks are cool I guess but from the sounds of it, the C code is a bit scary. If you had to build this, why not build it in Rust? At least it wouldn't be so terrifying from a security standpoint and you'd still get whatever performance is supposedly needed.
This code is more than a bit scary. This code is of very poor quality, is very poorly documented and has numerous (I'm pretty confident that they're exploitable, too) bugs.
It really isn't a project that I'd recommend to anyone, unless I wanted a shell on their machine. I'm quite distraught that it is getting this many (seemingly blind) upvotes.
I was thinking the same thing and then some. I ask a version of this as an interview question, and this doesn't pass my muster. That being said, I do think that implementing a URL shortener is a really good exercise, and I commend the author for that.
Yes. If there's a working buffer overflow anywhere in code (and there appear to be several here), you can turn that into a call out to a shell, and hand it some script code to perform.
> Wouldn't the process just die? It would say segmentation fault or something and return to the caller I would assume.
A segfault only happens when you try to access a virtual memory address not mapped to your process, or try to write to a page mapped to you but not mapped as writable.
Remember, this is C. There's nothing to stop you from writing past the end of some memory like an array as long as whatever memory you're writing into still belongs to you. If you write far enough, you'll eventually walk out of your mapped memory pages and trigger a segfault, but if you don't stray toooo far, you can overwrite some important data and absolutely nothing will complain. When people say C isn't a safe language, they mean it. C will let you get away with murder.
And it turns out that there's some very important data you can overwrite this way.
> How would you get a bash shell?
Let's say there's a function foo() that writes some data to a buffer on foo's stack, and I can control what data it writes (because it's data from a text box on a webpage, let's say). And foo() is buggy and doesn't validate that in all cases the data I control fits in the bounds of the buffer it writes to.
I can then overflow the buffer with my data, and take advantage of that to overwrite the return address of foo(), because the return address for the function happens to exist past the end of all the stack local memory for the function. When foo() returns, it will jump to the address I wrote in there, instead of where it was supposed to go back to. And as long as that address is in the process's mapped memory pages, again, nothing will complain.
To get a shell, as part of the overflow I either insert the binary data corresponding to the x86 instructions for something like a call to execve("/bin/sh", ...) and then have foo()'s return jump to the beginning of my instructions, or I cause the return to jump to some other code or library that will do that for me that happens to already be in place (there are more sophisticated versions of this exploit called Return Oriented Programming).
>Yes. If there's a working buffer overflow anywhere in code [...]
These days it's not _that_ straightforward with thing like ASLR, NX and heap hardening.
You need some sort of information leak for ASLR, then somehow start controlling R/EIP (which may prove difficult if there isn't interesting things on the heap nearby), write a ROP chain, and then pivot into something more useful (if you can't/don't want to ROP your way to a shell).
And now, reload this entire thread and check all the horrible bugs and security vulnerabilities this so called "feature" got you.
This is why you should never use C in networked code or when working with third-party data unless you absolutely bloody well have to: It's just too many ways to fuck up, and most programmers will.
No he didn't forget. The lower layers, yeah it probably will be in C (but it doesn't HAVE to be). But in the application layer, it probably shouldn't be.
If things like memory management are not the first priority when writing every line of code, then you shouldn't be using C, and that's really the only reason you should be using C, when there's a specific need for memory management.
Not surprisingly, this is exactly why the world has moved beyond C for the application layer, it's painful to have to think about that stuff constantly, so people just don't (this being an example).
Wasn't there something posted recently about urls that are too short are too easily guessed and that it's good to make sure that these short urls are longer?
102 comments
[ 34.5 ms ] story [ 69.6 ms ] threadWhat are the api endpoints? docs? Can you view a list of all shortened urls? Can you delete shortened urls?
Can you change the base domain of lon.gs?
EDIT: Very strange, the redirect now goes to a URL I didn't enter... (http://www.sadfasdfasfdasdfsadfasd.com)
YOURLS was popular for a while, and I tried it, but I was concerned with running a not very popular PHP app even on shared hosting. At least Wordpress gets decent attention. I was worried of people compromising my own YOURLS instance against me.
http://www.cvedetails.com/vulnerability-list.php?vendor_id=1...
No idea about equivalents for nginx, or something that could run on Github Pages (these would probably be the same, given that nginx doesn't have an equivalent to .htaccess out of the box).
I can see at least one buffer overrun dependent on database contents, and I wouldn't be surprised if there's public-facing vulnerabilities in this thing, but I don't want to spend another 5 minutes looking.
If the database could theoretically hold it according to its schema, the program should be prepared for the largest row size that is still technically legal. (Where "prepared" may simply mean that it throws a "This should never happen" error and aborts the request.)
Here's a contrived example:
At the user creation page the malicious user wants to get admin access.
They have their username be: admin' --
They do this because they are betting on a user named 'admin'.
Their password is: alwaysSanitizeInputs
So, a user is created named "admin' --".
Now, they go in to update their password.
The DB nicely pulls out their username, "admin' --", and it says:
So what happens? Well, the user "admin" now has a different password, and the Malicious user knows what it is. That's why you still need to sanitize.You get around it by parameterizing your queries, and not blindly trusting data that comes from the database.
Ideally a corrupt record only breaks the requests that read that record with an HTTP 500 or something similar.
https://github.com/riolet/longs/blob/master/longs.c#L238 looks like an example of a buffer overflow.
Etc.
EDIT: It also throws a whole bunch of warning when compiling. [moved this here from the first line after child post mention]
The compilation warnings was an additional remark, I should've phrased that post better.
[1] - https://github.com/riolet/longs/blob/master/wafer.c#L334-337
I don't have much experience with C outside of embedded systems, so my security practices in C are probably less than ideal for PC/server based code.
And when your URL shortener goes down, all links are dead. Shortening URLs is the last thing you want to do when you need to preserve pages that might go down. Caching is a better solution.
That's not the primary use-case for it[2][3], but we're happy to support people shortening URLs if they want to. The reason there's room for this is that our URL shortening function has no ads, no third party tracking, no bloat ... no dark patterns.
That's worth something to some people.
[1] Oh By, Inc.
[2] https://0x.co/examples.html
[3] https://0x.co/hnfaq.html
But it’s still a closed, proprietary database and there’s no way to decode an “Oh By Code” if the service goes down, which is the disadvantage number one you have with URL shorteners.
> The real utility are the easily recognizable codes, prefixed with "0x".
Why not using a prefix that helps differentiate your codes from hexadecimal numbers? How do you plan to get people to know your service if it’s not recognizable? Let’s say I put one of those codes on my business card, I still have to write somewhere that people should use 0x.co to access the content, which ruins the advantage of having just one code rather than a URL to my website (literally anyone knows how to open a URL in a browser).
Yes, that's correct. In this case, "Oh By" will continue running indefinitely. This is true for two reasons:
1) Building an extremely lightweight, ad free interface has a happy byproduct of ... being extremely lightweight. The infrastructure requirements for even a wildly popular "Oh By" are trivial.
2) Because Oh By is self-funded, there is no pressure from outside parties to break the pattern of an ad free, tracking free service with zero bloat.
If you're having trouble swallowing this, remember that rsync.net has been running continuously since 2001[1]. I think that's a credible track record[2].
[1] As a feature of JohnCompanies, the first VPS provider, and then as a standalone corporation in 2006.
[2] In fact, the rsync.net warrant canary turns 10 years old this year.
There's a 0% chance this will be running "indefinitely." You're just skirting the issue by making a promise you can't keep.
Well that's the real trick, isn't it ?
I'm happy to report that while Oh By itself does not display ads, we are a consumer of ads. We have a decent budget for sales and marketing.
I used to run a public one but it was getting abused for spam so I stopped.
http://www.archiveteam.org/index.php?title=URLTeam
(edit: and you can imagine how useful that's going to be once browsers start checking the Wayback whenever they see a 404... we're getting close to starting a trial of that with Firefox, with other browsers to follow.)
Here's the short url of http://news.ycombinator.com --> lon.gs/amk
Going to lon.gs/amk redirects me to an overstock.com address for a specific product.
The site doesn't appear to have been hacked, at least there's no affiliate link in the URL I was sent to. It just appears the site is broken.
Cool hacks are cool I guess but from the sounds of it, the C code is a bit scary. If you had to build this, why not build it in Rust? At least it wouldn't be so terrifying from a security standpoint and you'd still get whatever performance is supposedly needed.
It really isn't a project that I'd recommend to anyone, unless I wanted a shell on their machine. I'm quite distraught that it is getting this many (seemingly blind) upvotes.
https://www.reddit.com/r/C_Programming/comments/4p5ung/longs...
(After a few minutes of poking)
A segfault only happens when you try to access a virtual memory address not mapped to your process, or try to write to a page mapped to you but not mapped as writable.
Remember, this is C. There's nothing to stop you from writing past the end of some memory like an array as long as whatever memory you're writing into still belongs to you. If you write far enough, you'll eventually walk out of your mapped memory pages and trigger a segfault, but if you don't stray toooo far, you can overwrite some important data and absolutely nothing will complain. When people say C isn't a safe language, they mean it. C will let you get away with murder.
And it turns out that there's some very important data you can overwrite this way.
> How would you get a bash shell?
Let's say there's a function foo() that writes some data to a buffer on foo's stack, and I can control what data it writes (because it's data from a text box on a webpage, let's say). And foo() is buggy and doesn't validate that in all cases the data I control fits in the bounds of the buffer it writes to.
I can then overflow the buffer with my data, and take advantage of that to overwrite the return address of foo(), because the return address for the function happens to exist past the end of all the stack local memory for the function. When foo() returns, it will jump to the address I wrote in there, instead of where it was supposed to go back to. And as long as that address is in the process's mapped memory pages, again, nothing will complain.
To get a shell, as part of the overflow I either insert the binary data corresponding to the x86 instructions for something like a call to execve("/bin/sh", ...) and then have foo()'s return jump to the beginning of my instructions, or I cause the return to jump to some other code or library that will do that for me that happens to already be in place (there are more sophisticated versions of this exploit called Return Oriented Programming).
If want to read up on this: https://www.win.tue.nl/~aeb/linux/hh/hh-10.html
These days it's not _that_ straightforward with thing like ASLR, NX and heap hardening. You need some sort of information leak for ASLR, then somehow start controlling R/EIP (which may prove difficult if there isn't interesting things on the heap nearby), write a ROP chain, and then pivot into something more useful (if you can't/don't want to ROP your way to a shell).
This is why you should never use C in networked code or when working with third-party data unless you absolutely bloody well have to: It's just too many ways to fuck up, and most programmers will.
If things like memory management are not the first priority when writing every line of code, then you shouldn't be using C, and that's really the only reason you should be using C, when there's a specific need for memory management.
Not surprisingly, this is exactly why the world has moved beyond C for the application layer, it's painful to have to think about that stuff constantly, so people just don't (this being an example).
(they claim to have fixed this elsewhere in the thread, but I guess some of the "evil" URLs still work)
But that said, building a proper HTTP stack is not trivial.
If you want to use C language, then why not create Nginx module?
Nginx already solved the hard problems:
* HTTP parser
* Distribute work via event loop on multiple workers
* Useful load balancing strategies (not as great as HAProxy, but i am satisfied with it)
* Serious effort in dealing with CVE
* Widely used and battle tested
Here's a fine guide on how to write Nginx module in C: http://www.evanmiller.org/nginx-modules-guide.html