Ask HN: How willing are you to alpha/beta test a product (before release)
I'm thinking about a few subjects and a question came up: how willing are people to alpha/beta test a product before it's release to the general public?
I thought I'd ask Hacker News first.
9 comments
[ 3.2 ms ] story [ 27.5 ms ] threadYou have, by definition, launched publicly.
No one wants to debug someone else's broken crap. So deliver something worth an educated stranger's time with the caveat that it might be unpolished. People expect sites to get better with time.
An holographic display? Bring it on.
Bingo card generator? Not so much.
The most awesome bingo card generator? The same. Isn't about your product. Is about me.
I am of the opinion that this demographic enjoys poking at things for the sake of poking at things... but, as jp_sc, if it's totally orthogonal to our interests, our motivation drops dramatically.
Moreover, lsb makes an important point that bears repeating: no one* wants to debug someone else's broken crap. Someone released a beta link for a dating site within the last few days, and judging by the volume of criticism, it probably qualified as such. (No offense... it just wasn't ready. Some projects take time.)
*by and large
This worked out great, within a very short timespan we had thousands of beta users, who apparently agreed with us that our product was useful. With all the email we received from these people we were able to assess what kind of features people wanted added to our product (and which bugs really needed to be fixed right away) to be 'good enough' in their opinions. We mixed in the feedback with the improvements we had in mind already to release 8 more betas to incrementally get the product to a 'good enough' sweet spot that was being redefined/refined with every release.
Finally, we released a 1.0 (for pay) version of the product 5 monhts later.
Our application now has a large, very loyal and mostly happy group of customers and it won an Apple Design Award last month. I'd say our ideas about how we took it from beta to 1.0 worked out pretty well.
So, I'll echo what others have said here: Make something useful, release that, then work your ass off to turn it from something that's just useful (which is already a hard criterion to achieve) into something great.
If you just need 'testing': Hire someone to be a tester, or employ some friends to play with a private beta and give you feedback. We did this too before taking the beta public, and thereby prevented a lot of annoyances for our users and a ton of nearly identical emails in our support inbox for issues we really ahould've fixed before we released.
Even with this precaution in place, we still received multiple hundreds of emails a day just after the public beta release about other issues that we could have fixed before shipping but didn't yet know about. Releasing prematurely can not only turn away your earliest users, but it can also overwhelm you with a support burden you don't want to deal with when what you really should be doing is continuing development towards a solid 1.0.
For financial transactions or security concerns? Not so much. For creating work product I can evaluate and decide whether to use? Sure. Can I get my information back out, so that I've not lost all my time putting it into the product?
How easy is bug reporting?
Can I document and submit bugs quickly? I mean, I will put in the effort to be thorough, accurate, and concise in my descriptions. But can I do so without fighting the tracking software interface (and/or its host)? Can I easily and quickly track my issues, with update notifications so I don't have to remember to check status manually?
So, is the risk manageable, the effort streamlined, and the feedback responsive?
And, of course, does the product offer compelling features.
Making bug reporting easy seems under-rated to me. Many times, I've found a bug, to give up on reporting it because the process is too cumbersome. And response to submitted bugs motivates me, as well. I appreciate being able to help a communal effort. I don't expect a lot of time and effort returned unless I've found something significant; a status change in the tracker may be sufficient. But shoving bug reports into a black hole of status quickly becomes demotivating.