it fails at a lot of other basic things: I was unable to post anything because my firefox 52 is not supported to get a recaptcha challenge.
Please upgrade to a supported browser to get a reCAPTCHA challenge.
Alternatively if you think you are getting this page in
error, please check your internet connection and reload.
I could reload to infinity, this would not fix this broken website or change the useless error message.
Amazon rejects obvious non real browser user agents. This app tries to verify urls before putting them in the queue. I think they may have more anti scraper measures as well.
Well, I'm not exactly sure about how to do that. I could implement a list of all the major porn sites, but then people could still post porn URL using URL shorteners...
This is actually a good point, but I'd imagine people put down a porn website out of pure impulse when trying this service, so having their URL rejected with a friendly warning would certainly reduce the amount of porn spam. You could also block common URL shorteners.
Steven Black's hosts compilation [1] is a good start for what to disallow. For URL shorteners, you could disallow URLs that trigger an HTTP redirect (e.g. HTTP 301), or you could follow the chain of redirects and apply your rules to the terminal URL.
I used it a few times in a row, posting different ones of my bookmarks (for example, Isaac Asimov's Last Question and Last Answer stories), and at one point got my own link back.
Due to the low-quality links I always got before that I had first assumed it was just... garbage, but it seems it's just the users.
UPDATE: I've now tried about a dozen times (each time providing a link that isn't porn and contains informational content), and got 9 porn links, 1 racist site, and 2 actually interesting links, one of which seemed to have been added by an HNer: contentful.com
Tried it a few times, submitted some general interest links from my bookmarks, only to be redirected to scam sites :( Interesting idea, but it's unrealistic to expect good behavior from internet as a whole.
It could be useful to have a delay before redirect to be able to read the url and choose whether or not to go in it.
Yep. Just traded a link to a live stream of golden retriever puppies for someones kickstarter project and instantly thought "well I definitely should have expected that".
With noscript urlroulette is useless because it requires cloudflare, ajax.googleapis, google, gstatic and a few others for no valid reason apart supporting google surveillance and helping google for free with what their algorithm cant do (yet).
Once you've enabled cloudflare, anything can happen.
Yes, but they are no longer distributed to users. I thought that was the original question. I'm keeping them in the database to check for SPAM/multiple submissions mainly.
Of course the author is collecting everything submitted and does whatever he wants with it. Haven't you read the terms of service and privacy policy ? Probably because there are none which usually means you can safely assume that everything is collected, stored and exploited (or will be later).
Adding in a query param to bust the cache make it possible to submit the same site multiple times, i.e. example.com, example.com?1, example.com?a, example.com?123
I think the idea is to detect when different URLs contain the same content. That defends against duplicate entries like example.com/?foo and example.com/?bar (which are the same page).
Let me get this straight, you're using a monitoring tool for the website you work on, to check out how long individual people spend on it after you sent it to them via UrlRoulette? That's kinda neat (in a slightly disturbing way).
I wanted to see the user agent and request headers, since there's another comment about other sites being blocked.
I simply looked at basic web server logs. It's a beta site with hardly any traffic, so the only log entries were myself and those from the three users from Urlroulette. I was surprised to see the same IPs were still making requests when I went to close the terminal a while later.
Yeah, log diving is bread and butter, but reading your original comment it sounded like UrlRoulette had some kind of magical feature, where one submits a URL, then gets to see how long the stranger spends on the submitted website. Hence the clarification.
It would be an interesting feature actually, though it would probably require some annoying iframe tomfoolery.
Interesting idea, but it seems like it has the potential to get you into a lot of trouble. You don't know what kind of messed up stuff you are going to be redirected to.
Agreed this is a very annoying feature and IMHO unnecessary. But the good news is that the captcha didn't work and told me to get supported browser because firefox 52 apparently is not.
I suppose urlroulette is trying to be smart and fails, because recaptcha works on every other site in the same browser, though I still hate it with a passion.
Sounds interesting, but I don't want to try it out myself because I know that the internet is dark and full of terrors. Also, I don't want to get Rick-rolled.
It probably goes without saying, but you should definitely use some kind of sandboxed browser/session if you want to try this. One annoying URL someone could submit is superlogout.com, which uses logout CSRF to log you out of a whole ton of websites at once.
However, I couldn't get it to work in a private tab, because apparently the invisible captcha fails.
More like tricking you into interacting with a website in an unintended way, not only through GET parameters but through clickjacking and other tricks.
There's no such thing as "GET parameters". I assume you are referring to the query string but that's not necessary for encoding resource specific content. You could use the path structure (which is the common way these days), the domain name itself (eg different sub domains for different content), the user agent string, referrer, IP location (eg GeoIP). This is only just scratching the surface too as I've not even began to cover the client side rules one can apply to create a targeted page. Essentially anything that runs like an application can be dynamically generated to be specific to the visitor and since websites can be executed both at the server side and client side (eg Javascript) you're blindly placing you're trust on any URL.
Frankly, these days you can't even trust the domain name itself since punycode and data inlining create easy vectors for spoofing, and DNS poisoning attacks are still widely possible.
laumars, you seem to be antagonizing my comments and downvoting regularly. There are people with different perspectives here in HN, but I do not see you replying to each one of them.
Your only recent activity has been exclusively on my threads, with a tone not allowed on the HN comments section (personal attacks), baiting for responses so you can downvote them, since you can only downvote a reply once and there's a minimum karma requirement for downvoting so it's unfeasible to do it with fake accounts.
We just had a pointless argument thread 20 levels deep a few days ago (which was not the first one), where neither of us conceded to the other's point of view. A waste of time, HN karma, and I still think you were wrong.
You cannot down vote replies so it's not me who's been negative reping you. I also haven't been following you around HN nor posting personal attacks. I just said "all URLs are parameters" because your comment was weird and misleading (for reasons i elaborated in the other post i left yesterday).
HN is a technical discussion forum so there will be times when people will disagree with the technical points you raise; particularly when you continue to comment on technical topics you only partially understand (as seen here).
My advice to you would be to take a second to digest those counterarguments to check if they are informative rather than assuming every counterargument is a "personal attack". And if there's not enough data to understand their point then ask for clarification or an elaboration. I've learned a lot from the discussions on here just through reading those counterarguments and certainly am not too proud to admit when i have been wrong.
Previous to this, your 20 last comments were on another thread of mine. I could argue with you, but it is just a bait to downvote replies.
So you were randomly browsing HN news, and you found my comment, a non-top comment among hundreds or possibly thousands of comments, on a random post.
Then, you were not looking at the author, but rather at the comment, and you decided to reply to this one in particular. I am sure what I said strictly speaking, was incorrect, but not sure if this is the least correct of the comments in this topic.
Let me tell you that is unlikely enough to be considered intentional.
Then, stop mixing technical responses with insults. This is an insult:
> particularly when you continue to comment on technical topics you only partially understand (as seen here).
Just stop making excuses. This is the last response I give you on HN. I have implemented HTTP from the ground up I have nothing to prove to you.
The last 20 posts might have been on a thread of yours (I haven't checked) but only about half of that was actually replying to you - possibly less. And some of those posts were even defending you. A decision I since regretted when you turned psycho on me and demanded I refund you karma(!!!) despite being literally the only person in that thread that defended you. This was also why I gave up on that thread and stopped replying to you).
As I said before, I didn't check this post was by the same user as the other thread. I don't tend to pay much attention to user handles. Sure it probably seems a massive coincidence but coincidence is hardly proof of malicious intent.
> I have implemented HTTP from the ground up I have nothing to prove to you.
Then why even make such a weird statement as that? You can literally "implement HTTP from the ground up" with a shell one-liner. It still doesn't prove you understand the nuances of the HTTP protocol; let alone any have any grasp of web security (which was the real point you were discussing here when you blundered your way through HTTP protocol jargon).
> I could argue with you, but it is just a bait to downvote replies.
How would that even work when HN disables the option to down vote replies?
Honestly, I'm not out to persecute you. You just need to chill out a little on here if you want _other people_ to stop down voting you.
Is this some sort of infosec performance art? This seems extremely inadvisable for users or the person running the site. What if someone posts something illegal in the jurisdiction of the viewer or a link to a site with an embedded exploit?
I dunno...you have plenty of context from my point of view. It's a site with the word "roulette" in it that tells you what it's going to do: "Submit to be redirected to the previous visitor's URL!"
Well in GP you asked if there was any difference than clicking links on Reddit. I think you've found one: outside of /new (and even then rarely), posts on Reddit will not throw you to the wolves.
I feel like every time I leave a comment on HN that is 98% true, someone comes a long to helpfully inform me of the remaining 2%.
Sorry man, but that sort of thing just drives me nuts! Maybe I should paste a signature onto my posts. Something like "I estimate this comment to be 93 +/- 5% accurate. Please inform me of inaccuracies only if they fall out of those bounds." :)
Heh. I see what you mean, but that wasn't the intention. I'm not a reddit regular, but I know it's got plenty of dark places. I just don't get the uproar of OMG, something called "xyzroulette" that says it will send me to random places is dangerous. It doesn't seem deceptive or unexpected, that's the comparison I was trying to make.
Maybe I should have compared to outbound links on 4chan.
Well, that could happen on any website that you visit. IMO it does not make a difference whether you know the link before you click it or not, because you still don't know what you will get. But I am thinking about implementing some sort of virus/malware scanning.
The general idea is to return all the same headers as a get, without the body, so that's likely correct.
Unless there's code farther down that sends content length, etags, etc. You would want to send those too. Though I don't know that it really matters much.
Good point about Content-Length and ETag headers. So it sounds like one needs to do exactly the same amount of work to respond to a HEAD request as a GET request. Meaning the body still needs to be rendered (to calculate its length and hash). Just not include it in the response.
142 comments
[ 2.6 ms ] story [ 80.5 ms ] threadReturning 405 'Method Not Allowed' on a HEAD isn't unreasonable, and that is the issue we are seeing.
EDIT: did say GET on prior edit.
I think you meant "on a HEAD".
This is a really cool app though!
A fair amount of sites you don't want to visit have obvious urls. Or maybe you could scrape the title and display that as well.
[1] https://github.com/StevenBlack/hosts
http://pasted.co/b2dc3032
why even
Rule 44 - When the URL says "roulette", there's lots of porn abound.
Due to the low-quality links I always got before that I had first assumed it was just... garbage, but it seems it's just the users.
UPDATE: I've now tried about a dozen times (each time providing a link that isn't porn and contains informational content), and got 9 porn links, 1 racist site, and 2 actually interesting links, one of which seemed to have been added by an HNer: contentful.com
All the low-quality stuff: yes, that's the other users! :)
It could be useful to have a delay before redirect to be able to read the url and choose whether or not to go in it.
https://www.youtube.com/watch?v=lp1rHka_BkA
http://bit.ly/2oCL927
Proposed a funny video, some simple stuff like Cluster Hat & Gitea github
Once you've enabled cloudflare, anything can happen.
You probably want to allow repeats of domains, Like map links to something interesting.
I'm not sure what you mean here? I've not seen a site tell you "We don't use this query parameter" if you stick an additional param on there.
Pretty sea slugs: https://demo.gbif.org/occurrence/gallery?taxon_key=2292251
Bugs: https://demo.gbif.org/occurrence/gallery?taxon_key=797
Hummingbirds: https://demo.gbif.org/occurrence/gallery?taxon_key=5289
I got three blogs in return, one of which was about catching Pokémon. Fair swap...
I simply looked at basic web server logs. It's a beta site with hardly any traffic, so the only log entries were myself and those from the three users from Urlroulette. I was surprised to see the same IPs were still making requests when I went to close the terminal a while later.
It would be an interesting feature actually, though it would probably require some annoying iframe tomfoolery.
(Surely there are competitors to Hotjar, but as I am not that active in this field it is the only thing I have tried)
There must be something more you can do with this concept though.
However, I couldn't get it to work in a private tab, because apparently the invisible captcha fails.
Luckily I'm immune to this as I don't have an account on any of these websites but one and I only log there once every 2 years for about 5 minutes.
https://youtu.be/G0u7lXy7pDg?t=18m13s
1) There are URL expanders with API to battle URL shorteners, here are some I found with quick googling:
http://www.linkexpander.com/api
https://unshorten.me/api (The API is limited to 10 requests per hour for new short URLs - too slow for you, probably)
http://pro.urlex.org (paid)
2) You could disallow URLs which are detected with 2 or more engines by VirusTotal:
https://www.virustotal.com/en/documentation/public-api/
https://en.wikipedia.org/wiki/Clickjacking
Frankly, these days you can't even trust the domain name itself since punycode and data inlining create easy vectors for spoofing, and DNS poisoning attacks are still widely possible.
The only safe way to browse the web is simply not to.
Your only recent activity has been exclusively on my threads, with a tone not allowed on the HN comments section (personal attacks), baiting for responses so you can downvote them, since you can only downvote a reply once and there's a minimum karma requirement for downvoting so it's unfeasible to do it with fake accounts.
We just had a pointless argument thread 20 levels deep a few days ago (which was not the first one), where neither of us conceded to the other's point of view. A waste of time, HN karma, and I still think you were wrong.
HN is not the Monty Python argument clinic: https://www.youtube.com/watch?v=kQFKtI6gn9Y
HN is a technical discussion forum so there will be times when people will disagree with the technical points you raise; particularly when you continue to comment on technical topics you only partially understand (as seen here).
My advice to you would be to take a second to digest those counterarguments to check if they are informative rather than assuming every counterargument is a "personal attack". And if there's not enough data to understand their point then ask for clarification or an elaboration. I've learned a lot from the discussions on here just through reading those counterarguments and certainly am not too proud to admit when i have been wrong.
So you were randomly browsing HN news, and you found my comment, a non-top comment among hundreds or possibly thousands of comments, on a random post.
Then, you were not looking at the author, but rather at the comment, and you decided to reply to this one in particular. I am sure what I said strictly speaking, was incorrect, but not sure if this is the least correct of the comments in this topic.
Let me tell you that is unlikely enough to be considered intentional.
Then, stop mixing technical responses with insults. This is an insult:
> particularly when you continue to comment on technical topics you only partially understand (as seen here).
Just stop making excuses. This is the last response I give you on HN. I have implemented HTTP from the ground up I have nothing to prove to you.
As I said before, I didn't check this post was by the same user as the other thread. I don't tend to pay much attention to user handles. Sure it probably seems a massive coincidence but coincidence is hardly proof of malicious intent.
> I have implemented HTTP from the ground up I have nothing to prove to you.
Then why even make such a weird statement as that? You can literally "implement HTTP from the ground up" with a shell one-liner. It still doesn't prove you understand the nuances of the HTTP protocol; let alone any have any grasp of web security (which was the real point you were discussing here when you blundered your way through HTTP protocol jargon).
> I could argue with you, but it is just a bait to downvote replies.
How would that even work when HN disables the option to down vote replies?
Honestly, I'm not out to persecute you. You just need to chill out a little on here if you want _other people_ to stop down voting you.
Also you have zero context and no hive-mind to check the link for you ahead of time.
I knew it was likely to throw me to the wolves.
There are plenty of subreddits where outbound links are dangerous or offensive.
Even one where they are almost guaranteed to have a 50/50 chance of being so.
https://www.reddit.com/r/FiftyFifty/
Sorry man, but that sort of thing just drives me nuts! Maybe I should paste a signature onto my posts. Something like "I estimate this comment to be 93 +/- 5% accurate. Please inform me of inaccuracies only if they fall out of those bounds." :)
Maybe I should have compared to outbound links on 4chan.
"Use at your own risk, the internet is a dangerous place!!"
They are doing a HEAD request to see if the url is valid, and if it returns anything other than 200 OK, they reject the url.
Edit: I went with something like this [1] for now. It worked to solve this problem. I can adjust later as needed.
[1] https://github.com/shurcooL/home/commit/b4a601ff7a3752feb291...
Unless there's code farther down that sends content length, etags, etc. You would want to send those too. Though I don't know that it really matters much.
Good point about Content-Length and ETag headers. So it sounds like one needs to do exactly the same amount of work to respond to a HEAD request as a GET request. Meaning the body still needs to be rendered (to calculate its length and hash). Just not include it in the response.