Is the author proposing an open-source tool for registration automation/scraping, which would compete with LastPass, 1Password, etc? Or open-source infrastructure that could be used by all such services?
Hey. Author here. Yes the tool would be open source and automate the process of registration. Alongside registration, it will borrow ideas from bugmenot where logins are ranked according to whether they work or not. The only difference being that machine learning and other means are used to rank instead of an upvote system which is mostly unreliable.
Keep in mind, there are inherent problems like T.O.S breaching so the tool would have to deal with them the right way, and through the right channels. Perhaps a standard could be implemented and sites could 'opt in' to having the skeleton key used on their site. So yeah - like open source infrastructure too.
Would the proposed system use shared accounts, or automated new account registration for the user?
If the former, I think the opt-in is a dead end. If the websites thought that was good for them, they'd enable anonymous use, and it even might be illegal for them to cooperate (e.g. they would be breaking COPPA by offering a way for under-13 to use the site without following the proper steps).
It would use both shared accounts, and auto registration, which depend on how complex the registration system is and what requirements it has. For example, on sites which have lots of traffic it is preferable to have a swarm of newly registered accounts arriving all the time because of unreliable users of the account deleting it, changing the password etc. On less traffic'd sites it would be preferable to have a few small accounts. I've observed this on bugmenot...
After reading the original post it's still not clear to me why you would want this tool - Would you mind explaining the motivation behind it ?
As an end-user of sites I don't find registration a particularly onerous or annoying problem - signup, login, move along.
As an operator of a few different sites the concept of users spamming the crap out of my registration system using automation is an annoyance I could do without.
Seems more like an edge case. If you look at bugmenot, the system works beautifully, and I am still amazed at how well it works, and how long the logins work despite the system being a bit sneaky and breaching the T.O.S on sites where multiple accounts and account sharing are forbidden. I wouldn't propose the system if it was going to harm the web in some way, and I said before that we can standardize it so that webmasters can allow a skeleton key to be used. Sort of like try-before-you-buy mechanism. Most sites do not have the luxury of some anonymous-user function where we can login instantly and start seeing walled garden information
Well if it's a service then it isn't really different than most SSO providers out there. Yes it's technically different but the UX is virtually the same (and even easier with an SSO). You sign up once manually, then have to make several easy steps usually revolving just authorizing the 3rd party application to use your account.
Now i understand why many sites don't want to use SSO's, they lose control over their user base and get much less information from a registration via SSO than they would via their own registration form, and always the SSO providers (e.g. Twitter and LinkedIn) can always say you know what we don't want to provide that service any more, and in many cases you can be left with no ability to recover a large part of your user base.
That said I haven't seen many sites that Chrome cannot populate virtually their entire sign up page from memory, and like most stuff in Chrome it's also (can be) shared between devices as long as you are signed into one. Heck not that i would advise it but Chrome can even store your credit card number "securely" although that option i believe does not replicate between devices (never used it myself, not crazy:)).
Yes sign up processes can be annoying, especially when they obviously unnecessary, but from experiences the sites that put everything behind a signup wall rarely have much content to offer in the first place, since sites that actually want to monetize their user based tend to have very easy signups in order to reduce drop out rate. Heck just look at payday loaners, their sign up is as short as possible to get some info about you for a credit check and that's it, they don't want people to drop off before they get a loan and every step increases the chance that they'll figure out that taking one at 123514% APR isn't the best solution for their financial shortcomings.
"Heck not that i would advise it but Chrome can even store your credit card number "securely" although that option i believe does not replicate between devices (never used it myself, not crazy:))."
If my OS or Chrome is compromised I have bigger things to worry about than my credit card number, for which charges can easily be reversed...
Yes, but there's a compromise and a compromise, say there is a bug in Chrome that allows some one to exploit it in order to dump all of the auto-complete history including your CC but nothing else.
Not everything results in a persistent exploitation of a system.
And even in such cases when the exploitation is persistent it's still a question of how much can they get and in what time window since you might detect it and remediate the situation.
I've seen some sneaky tactics that some sites used to get details from auto complete functions of browsers, yes chrome asks you if you want to submit that data, and asks you specifically for your CC but if you mask some fields such as address and phone number (Chrome detects if it's hidden, and to some extent when its not visible due to other factors so it needs to be just cleverly disguised in rendering) when some one just auto-fills say their email and username it might also leak personal information, and if people are not too careful also they're payment data (which i believe Chrome only auto-fills on sites you visited before, and on secure connections).
I personally only have to input my CC details online every couple of months since i use PayPal or Amazon Checkout for about 90-95% of my purchases, so i rather not have my CC data stored on my machine if i can avoid it. Yes if it's compromised they can eventually get it, but it's still better than it being stored in a place where every malware on the planet knows to look for it.
The #1 reason I prefer mobile apps to web apps is because even those that require sign in will remember my login permanently. The entire rest of the web (with rare exceptions) all seems to believe that it's acceptable to require re-login over and over and over. I've drastically reduced the number of sites I use just to end the madness, even with 1Password. How is this still not solved in 2015?
Hey author here. All valid points. I see you made a tradeoff there. So you asked how is this still not solved in 2015? Did you mean rhetorically, as in, you use your phone for logging in so your problem is solved, or you want to see a skeleton key system built?
How long until someone learns to use replay attack to hijack sessions from a mobile app?
I'd say about half, probably more, of the web sites I log into are able to hold the session for an extended period of time. A few are very aggressive about expiring cookies and often it's because my IP address changes. Those sites are also ones that deal with frequent spamming or DDoS.
On the other hand, frequently having to login helps me remember my passwords. I can tell you right now I probably won't get into HN on the first try because I haven't had to type the password here for over a year.
Many websites don't seem to have mandatory session expiry. I have Firefox set to clear everything except cookies and site preferences when I close my browser, and then set cookies to "until I quit Firefox", with exceptions for sites I want to stay logged in to (reddit, HN, few other non-critical but frequently visited sites) set to "allow".
Self-destructing cookies to purge cookies from closed tabs after a timeout is also a useful extension.
>"The introduction of a National Postcode System, known as "Eircode", is planned to take place in Summer 2015."
>"An Post did not introduce automated sorting machines until the 1990s. By then, the optical character recognition (OCR) systems were advanced enough to read whole addresses, as opposed to just postcodes, thereby allowing An Post to skip a generation. Consequently, mail to addresses in the rest of the state does not require any digits after the address."
Aren't most website storing long-term cookies when the user check the "remember me" function? Also, I think that the most difficult part of an automatic signup system would be captchas.
This is something I have thought about, but I've observed a great many websites that don't use a captcha because of whatever reason. All that's needed here is to bulk-solve captchas as they are needed. Not necessarily in a nefarious manner, but simply to isolate/silo the captcha solving process away to another system. There could a separate H.I.T (Human Intelligence Task) system similar to Amazon's Mechanical Turk program where the 'hard problem' of captchas scales down to a more manageable problem.
Depending on the size and scope of the skeleton key system, there could even be (very willing) people ready to solve them in exchange for a small fee. There's a startup if I saw one. Users of the skeleton key system could go premium if they want to bypass captcha systems and a portion of the money goes to the captcha-solver. Probably not a new idea, as I've seen many such nefarious systems in place for spam, except this would be legitimized!
you want to bulk solve captcha's, in a non nefarious manner? but do you realize that bulk solving is nefarious? captcha is to prove you are human, not a bot or script.
Except for the fact that humans are solving them, only getting paid for solving them and rendering said captchas effectively redundant. The key term here is isolation, where rather than captchas serving to hurt flow states, they are isolated away and offloaded more comfortably to the solvers who are happy to solve them in return for a reasonable fee. Google's new system is a bit more robust however and uses fingerprinting. One could argue that fingerprinting is more nefarious than letting people have a small income from captcha solving. On the subject of Google's fingerprinting http://blog.higg.so/2015/02/24/googles-new-captcha-security-...
you think the fact that humans solving them in an automated fashion, instead of a computer doing it, is what makes all the difference? Rendering the capthas redundant is the problem.
Maybe you just need an app that signs you up for sites.
Please refer to my comment on Dysprosium's post for my answer on this. Also I may add that it is not a system for spammers, but a system like bugmenot where we can bypass the often arduous process of registration. I happen to have RSI (Repetitive Strain Injury) on my wrist and registration forms are the final boss of the internet so although a system would be nice for myself, I can also see a huge need for others to use it.
Have you considered looking into Lastpass? There's a really great feature called "Form Fills" which allows you to save your personal information to a profile, and then filling out registration forms is as simple as right clicking on the first field -> clicking "Form Fill".
Yeah I tried Lastpass. I love that feature, but it is not a silver bullet and it often gets things wrong, populating the wrong fields and also still requiring some manual effort. I use Lastpass but I was spooked last year when their servers were (briefly) taken offline. Be careful of hosted services with all your mission critical passwords stored there. I now do local copies of LP when I get the chance.
Building an easily-automated registration system that's "not for spammers" is like building a back door into an encryption scheme that's "not for foreign state-sponsored hackers".
Automation is a small part of the system and not the main concern. The main concern is making the experience of surfing the web less harrowing for surfers and to make it more smooth. There are many walled garden sites arbitrarily locking away information and that is counter intuitive to what the web is about, which is free access to information. In the true nature of the web, there would be mechanisms in place to unlock information, like a web browser is expected to unlock information. People's use of fake useragents that spoof a Googlebot to access Quora being a perfect example.
I'm looking for a way to submit new links to a list of computer employers that I'm building. I face the problem of spammers, specifically. One idea I have in mind is for the registration form to be at a new url every time, generated for that specific registration, with a significant time delay before my server coughs up the URL.
If I do it right, to a live human it will appear that the internet is just a little slow at the time.
This seems like a solution for a problem that mostly doesn't exist anymore. I rarely encounter sites that require registration merely for reading. Many newspaper sites used to do that, but now they're mostly open, or paywalled.
On sites where I post or buy stuff, why would I want shared accounts? Maybe I can see it for software support sites. But even there, account reputation can be important.
I thought of that too, and it would not be preferable to have shared logins, or an automated system in place for registering on e-commerce sites, or sites where money is moving in them. One could argue that a large portion of websites have money moving through them in one form or another, but I don't have the numbers / stats for that. This would be more for general purpose sites that have wall-gardened information, and also for sites that have arbitrary registration forms which exist for all the reasons (or lack of reasons) I outlined in my post.
Fair enough. Maybe some examples would help me understand.
I do get that registration can be a pain. But what I'd want wouldn't be a skeleton key, but rather a personal master key. Logins using Facebook and Google have become common.
But, being a privacy geek, I don't have accounts. And I definitely don't do cellphone authentication, because that is nontrivial to anonymize. However, I would want two-factor authentication. That's the tough problem, I think.
The more I think about this "problem" too, the more it goes back to the question of why do these sites need my information, and subsequently what way can they verify my identity?
We need a better web of trust where another service can verify the identity of me as a user. Already with the likes of Chrome, much of my logins are already auto-filled meaning the browser is already happy with my identity, why can't part of this be used for signups. We have second-factor authentication, again why can't we leverage this to verify I am the real person. Between the browser, a third party service and the receiving service we can securely verify an identity without the need for passwords, email addresses and real names.
In order to use Facebook or Twitter to find information on those services, one must create a new account and divulge all sorts of personal information.
accounts that are used by groups are harder to track and pinpoint, and their site usage will likely be too random to glean meaningful data from them.
Yes, sites have different flows for signup. This is not a "problem". Different sites want different things, or have different needs. Some with have Captcha, others will require CC, or what have you.
Yes, there are some sites that require login and are just being asshats for it. And for those, bugmenot is good.
But for others, there is a need for signup. And that shouldn't be swept under the rug.
I think a better approach could be to push for more usability and accessibility of the forms. Because thats something I am all for.
Your "ultimate solution" will just become another on top of the "ultimate solutions" listed as the services you indicated on your post.
Your idea seems too narrow to address a mass-market.
The tools that exist are already serving niches:
- Bugmenot - to bypass signups where something important is needed behind the signup-wall but probably not an account you'd use for amazon.com
- Quick signups = social login . Most people are content to connect to sites via Google/Facebook login, which avoids the entire signup process
- Lastly, don't forget OpenID - it died recently and was an attempt to unify the login process. Probably exactly the use-case for you.
Truth be told, site-makers have signups because they can and do so because they want to capture data from users for "important" advertising, user-data-selling, etc.
So long as users are happy to sacrifice some data for a 'free' service, this is the model the internet will continue to thrive on and signups will continue.
If you want to go ahead and try to solve the problem for yourself, it may be worth it. For the average user though, this is another non-problem in their quest to use x-social-cool-new-app that requires me to just login via Facebook to so I can "share" stuff.
1. The NASCAR problem. Listing out every popular service does not guide the user through the login process.
2. The lost NASCAR problem. I know I signed up with one of these services but which one for this site?
3. The abandoned service problem. How do I access this site after my identity provider closed shop or the service I was using decided to stop supporting OpenID?
4. The redirection dialog problem. What do all these dialogs mean and what access am I granting to who?
5. The phishing problem. The default experience lends itself to
teaching people to type secrets into popups / redirects coming from other sites.
6. Nobody wants to be a URL; email is king here. The support for email came late in the game.
7. The double registration problem. I signed up with OpenID which gives the site my basic profile but I still have to "finish" registration by giving more information.
There are many other issues but those are the big ones to me.
OpenID is still a thing (its hard to kill a protocol). While most OpenID providers have gone away, indieauth.com will allow you to delegate your domain name to it.
...and you can easily be your own provider. The main issue is that pretty much nobody supports it now, I haven't seen a 'Login with OpenID' button for a while.
OpenID has gone the same way as XMPP, a good open and decentralised protocol lost to proprietary social networks.
What a weird thing to see on HN: the top-rated comment is telling someone that they can't do something, and that it's a bad idea. Perhaps this is the best reason to actually make this idea happen?
52 comments
[ 6.8 ms ] story [ 52.9 ms ] threadKeep in mind, there are inherent problems like T.O.S breaching so the tool would have to deal with them the right way, and through the right channels. Perhaps a standard could be implemented and sites could 'opt in' to having the skeleton key used on their site. So yeah - like open source infrastructure too.
If the former, I think the opt-in is a dead end. If the websites thought that was good for them, they'd enable anonymous use, and it even might be illegal for them to cooperate (e.g. they would be breaking COPPA by offering a way for under-13 to use the site without following the proper steps).
As an end-user of sites I don't find registration a particularly onerous or annoying problem - signup, login, move along.
As an operator of a few different sites the concept of users spamming the crap out of my registration system using automation is an annoyance I could do without.
Now i understand why many sites don't want to use SSO's, they lose control over their user base and get much less information from a registration via SSO than they would via their own registration form, and always the SSO providers (e.g. Twitter and LinkedIn) can always say you know what we don't want to provide that service any more, and in many cases you can be left with no ability to recover a large part of your user base.
That said I haven't seen many sites that Chrome cannot populate virtually their entire sign up page from memory, and like most stuff in Chrome it's also (can be) shared between devices as long as you are signed into one. Heck not that i would advise it but Chrome can even store your credit card number "securely" although that option i believe does not replicate between devices (never used it myself, not crazy:)).
Yes sign up processes can be annoying, especially when they obviously unnecessary, but from experiences the sites that put everything behind a signup wall rarely have much content to offer in the first place, since sites that actually want to monetize their user based tend to have very easy signups in order to reduce drop out rate. Heck just look at payday loaners, their sign up is as short as possible to get some info about you for a credit check and that's it, they don't want people to drop off before they get a loan and every step increases the chance that they'll figure out that taking one at 123514% APR isn't the best solution for their financial shortcomings.
If my OS or Chrome is compromised I have bigger things to worry about than my credit card number, for which charges can easily be reversed...
And even in such cases when the exploitation is persistent it's still a question of how much can they get and in what time window since you might detect it and remediate the situation.
I've seen some sneaky tactics that some sites used to get details from auto complete functions of browsers, yes chrome asks you if you want to submit that data, and asks you specifically for your CC but if you mask some fields such as address and phone number (Chrome detects if it's hidden, and to some extent when its not visible due to other factors so it needs to be just cleverly disguised in rendering) when some one just auto-fills say their email and username it might also leak personal information, and if people are not too careful also they're payment data (which i believe Chrome only auto-fills on sites you visited before, and on secure connections).
I personally only have to input my CC details online every couple of months since i use PayPal or Amazon Checkout for about 90-95% of my purchases, so i rather not have my CC data stored on my machine if i can avoid it. Yes if it's compromised they can eventually get it, but it's still better than it being stored in a place where every malware on the planet knows to look for it.
I'd say about half, probably more, of the web sites I log into are able to hold the session for an extended period of time. A few are very aggressive about expiring cookies and often it's because my IP address changes. Those sites are also ones that deal with frequent spamming or DDoS.
On the other hand, frequently having to login helps me remember my passwords. I can tell you right now I probably won't get into HN on the first try because I haven't had to type the password here for over a year.
Self-destructing cookies to purge cookies from closed tabs after a timeout is also a useful extension.
>"The introduction of a National Postcode System, known as "Eircode", is planned to take place in Summer 2015."
>"An Post did not introduce automated sorting machines until the 1990s. By then, the optical character recognition (OCR) systems were advanced enough to read whole addresses, as opposed to just postcodes, thereby allowing An Post to skip a generation. Consequently, mail to addresses in the rest of the state does not require any digits after the address."
Depending on the size and scope of the skeleton key system, there could even be (very willing) people ready to solve them in exchange for a small fee. There's a startup if I saw one. Users of the skeleton key system could go premium if they want to bypass captcha systems and a portion of the money goes to the captcha-solver. Probably not a new idea, as I've seen many such nefarious systems in place for spam, except this would be legitimized!
Maybe you just need an app that signs you up for sites.
Why the fuck would any service provider want to make it easy for spammers to automate mass registrations as the author seems to want?
It's literally the reason CAPTCHA was invented.
Given that, why wouldn't website providers take active steps to block this?
I'd say browser auto compete solves 85% of this issue. The last 15% his the law of diminishing returns pretty quickly.
If I do it right, to a live human it will appear that the internet is just a little slow at the time.
On sites where I post or buy stuff, why would I want shared accounts? Maybe I can see it for software support sites. But even there, account reputation can be important.
I do get that registration can be a pain. But what I'd want wouldn't be a skeleton key, but rather a personal master key. Logins using Facebook and Google have become common.
But, being a privacy geek, I don't have accounts. And I definitely don't do cellphone authentication, because that is nontrivial to anonymize. However, I would want two-factor authentication. That's the tough problem, I think.
We need a better web of trust where another service can verify the identity of me as a user. Already with the likes of Chrome, much of my logins are already auto-filled meaning the browser is already happy with my identity, why can't part of this be used for signups. We have second-factor authentication, again why can't we leverage this to verify I am the real person. Between the browser, a third party service and the receiving service we can securely verify an identity without the need for passwords, email addresses and real names.
accounts that are used by groups are harder to track and pinpoint, and their site usage will likely be too random to glean meaningful data from them.
Yes, sites have different flows for signup. This is not a "problem". Different sites want different things, or have different needs. Some with have Captcha, others will require CC, or what have you.
Yes, there are some sites that require login and are just being asshats for it. And for those, bugmenot is good.
But for others, there is a need for signup. And that shouldn't be swept under the rug.
I think a better approach could be to push for more usability and accessibility of the forms. Because thats something I am all for.
Your "ultimate solution" will just become another on top of the "ultimate solutions" listed as the services you indicated on your post.
Your idea seems too narrow to address a mass-market.
The tools that exist are already serving niches:
- Bugmenot - to bypass signups where something important is needed behind the signup-wall but probably not an account you'd use for amazon.com
- Quick signups = social login . Most people are content to connect to sites via Google/Facebook login, which avoids the entire signup process
- Lastly, don't forget OpenID - it died recently and was an attempt to unify the login process. Probably exactly the use-case for you.
Truth be told, site-makers have signups because they can and do so because they want to capture data from users for "important" advertising, user-data-selling, etc.
So long as users are happy to sacrifice some data for a 'free' service, this is the model the internet will continue to thrive on and signups will continue.
If you want to go ahead and try to solve the problem for yourself, it may be worth it. For the average user though, this is another non-problem in their quest to use x-social-cool-new-app that requires me to just login via Facebook to so I can "share" stuff.
Good luck!
1. The NASCAR problem. Listing out every popular service does not guide the user through the login process.
2. The lost NASCAR problem. I know I signed up with one of these services but which one for this site?
3. The abandoned service problem. How do I access this site after my identity provider closed shop or the service I was using decided to stop supporting OpenID?
4. The redirection dialog problem. What do all these dialogs mean and what access am I granting to who?
5. The phishing problem. The default experience lends itself to teaching people to type secrets into popups / redirects coming from other sites.
6. Nobody wants to be a URL; email is king here. The support for email came late in the game.
7. The double registration problem. I signed up with OpenID which gives the site my basic profile but I still have to "finish" registration by giving more information.
There are many other issues but those are the big ones to me.
OpenID has gone the same way as XMPP, a good open and decentralised protocol lost to proprietary social networks.
https://stackexchange.com/users/login https://bitbucket.org/account/signin/ http://openid.yahoo.com/ still responds, surprisingly. The wordpress openid plugin is still around.