In the past, I've done an invite-only beta to limit the exposure of issues with my code while in a QA-phase (minimize embarrassment) and to retain control over the load/expense of hosting. I suppose that others could be going with the "keep a line to insinuate exclusivity" approach.
At the risk of making a very elaborate "this" post, I agree with all above points.
I've mainly used invitation-only to minimize load or at least to get a better idea of where the pinch points were in where scaling is going to be tricky. It's nice to be able to say "We're operating just fine with 100 logged in users. What happens if we have 1,000 logged in users?" and be able to control that to some degree. Giving each user a couple of invites is going to grow the population (you hope) but not open up the firehose completely. If you don't know how you're going to scale to 1,000 users, you might not want a million users just yet.
How about using Open-Source tools such as Jmeter to run performance test and identify the bottle neck areas. We have done that with our site http://www.jackpotbuddy.com. Initially were were getting around 25 seconds page load time for 1000 users.
We were able to reduce that to <5 seconds after identifying and fixing the issues
That's probably a really good suggestion, I can't completely say. In my experience, load-generating tools give completely differing results than actual user patterns, but it's been awhile since I've used any with any real expectation.
Obviously it's great to test more obvious things, but I would use something like that before real user testing, not in lieu of.
How have you seen JMeter to compare in this? I haven't ever used it directly.
There is a flawed assumption that opening up the floodgates will flood the village. There is a very small coterie of people that can, and will, use their influence to maintain interest in a project over time. For most of us, no one really cares enough about what we're doing anyway. So, by turning away one potential user b/c they don't have an invite code, you are hurting yourself...maybe more so than a buggy, but still usable UX might.
That being said, my current startup project http://www.browsemob.com has an invite code to get into our beta, even though I put it in our signup video and all over the net.
That's awesome! Do what I say, not what I do! I completely agree though. I think an invite system, in theory, lets you build a sense of exclusivity. The velvet rope effect - people waiting to get in the hot night club. But in the end very few sites can really demand that kind of attention. We are using an invite code and launch page on our project - http://neufit.com and are allowing in-game invites for those 'lucky' enough to get under the rope! Hope it works! Just signed up for browsemob, btw!
6 comments
[ 2.9 ms ] story [ 22.9 ms ] threadI've mainly used invitation-only to minimize load or at least to get a better idea of where the pinch points were in where scaling is going to be tricky. It's nice to be able to say "We're operating just fine with 100 logged in users. What happens if we have 1,000 logged in users?" and be able to control that to some degree. Giving each user a couple of invites is going to grow the population (you hope) but not open up the firehose completely. If you don't know how you're going to scale to 1,000 users, you might not want a million users just yet.
We were able to reduce that to <5 seconds after identifying and fixing the issues
Obviously it's great to test more obvious things, but I would use something like that before real user testing, not in lieu of.
How have you seen JMeter to compare in this? I haven't ever used it directly.
That being said, my current startup project http://www.browsemob.com has an invite code to get into our beta, even though I put it in our signup video and all over the net.
What can I say, my hypocrisy has no bounds!