Enlisting the help of friends and family can only get you so far, so how do you gather quality feedback pre/post-launch, without shelling out for a professional agency to do the testing?
What's your goal? Do you want to test whether your update works correctly on many devices? Do you want to get ideas for new features? How many users do you have?
I have about 10,000 active users. I don't do much testing. I test myself on an Android device and an iPhone. Since my app is built with Ionic and based on Cordova, it works anyways and real crashes are not to expected.
Feedback is easy: I just wait until people write me an email. It's funny: Sometimes, you don't hear about a feature request or some issue for months. Then 2-3 people write you within a day. Then nothing again.
Back when I started, I implemented a lot of features. Now I basically reject almost every feature request, unless it comes up extremely often in emails. I sometimes take notes, but most of the time, I type a quick reply, thanking for the suggestion, archive the email and only try to get a feeling for what people want.
Obviously, this has its limits, but since I know my core users and what they want, it works for me.
If there's a serious bug, people will let you know soon enough.
Post-launch is mainly vetting that new features makes sense to users and catching UX faux pas, instability or even crashes, etc. before opening up to a wider audience, and possibly scores of bad reviews.
I was thinking about just opening up on the app store as well, and just sit back and wait for reviews or emails, but that seems risky, since potentially bad reviews are forever. For a paid app at least, that can have a pretty serious impact on possible uptake. Having at least some level on confidence, that the app is usable and stable before general availability, is the safe play I think. To that affect, closed testing seems the only viable option.
I've used UserTesting.com for previous tests. If your product affects a lowest-common-denominator user or you are testing mainly UX/interaction, I'd recommend it. At that time I paid around $20 per usability test (I bought a packet of multiple sessions). Testers recorded using my product for 5 to 10 minutes and it was great watching them stumble over what I had designed. They were also trained to be over-communicative and think aloud, which really helped. In total I did four or five of them before I had my first release (since it was still somewhat expensive, I tried to ration my tests by fixing all previous UX bugs before issuing another one). It's been a long time, but if I recall correctly, a few times the testers weren't tech-competent enough to actually do the test, but when I asked UserTesting for a credit or refund to re-do the test because the tester was "bad", they provided one without hassle.
You need to be make it as easy as possible to run your user test, because a tester will give up on roadblocks that take more than 60 seconds to solve.
I was developing a language learning product, and so needed testers who were interested in language X. In the end, the people I found using the filter [Interested in X language, native English] weren't actually interested in language X. Rather, they came across as people who had accepted the filter to make an extra buck.
However, it was good enough for a random internet trial. If I wanted better results I had to do it in person (which I also recommend).
I usually hang out in the kitchen at our shared office space and ask people. If that's not enough, then I'll start visiting friends' offices for team breakfast or post-work beers and do the same.
In the past (when I was still a solo developer), after being 90% confident the code won't completely break the environment or do something screwy with user data, sticking the code in production and funneling 50% of the requests to it while watching NewRelic.
There are no better user testers than your users, and the risk is only moderate. I could always pull the server down and restore the previous version while trying to fix whatever I broke in the deployment.
But if you aren't doing SaaS, user testing as a solo developer is not really feasible. The best user testing I can think of that doesn't "cost" is a Beta/RC Program for your users to volunteer to get potentially broke software for a discount or just early feature releases. For this to work though, you have to have enough users.
I agree, it's easier to do for web apps since you don't have to contend with "app store reviews", that can potentially leave a lasting blemish on your product. So far I've done it as you describe as well and it's the "getting enough users" part that's a pickle. If you get X users on to sign up for a beta, only a fraction of those will actually ever install the app and give it a spin and again, only a fraction of those will give you any real feedback to work from. Finding real testers without breaking the budget, of your (maybe zero-profit) app is the challenge.
For web apps I’ve always used Mechanical Turk and paid each user $10 vs. just a dollar or less. I made each tester download a screen record plug in for chrome, hit record and then go through all the steps in the test. Once finished they emailed me the recording.
The above is what you get from usertesting.com and other sites but pay up the nose($50 a test) vs. $10 or less dollars per test.
If it’s a downloadable app Just spin up a web UI/UX and have the turkers follow same steps but have them change the browser to say iPhone 6/7/8 view.
I don't understand this, why would you want people to test something that's already predefined? You could automate that.
You want real users that will use the app for real things. Blind testing from a list seems kind of pointless.
The goal is to detect usability issues. Your features should have user stories associated to them, come up with few scenarios around these (e.g. book a flight, cancel your flight from the landing page...). Then test your scenarios with multiple users/personas.
This will allow you to detect UX/usabilities issues, and prioritize update based on how critical the issue is.
I'd say it depends on the sort of application you're developing.
In the next couple of weeks I'm due to release a boxing game (shameless plug: leatherthegame.com), and other than myself my only tester has been my cousin.
Having worked on the project for over 3 years I'm aware that there are areas where I "can't see the wood for the trees", but I am confident in my own knowledge, and only wanted an external viewpoint to see how a noob would see things from afresh (both in terms of not having used the app before and also him being only a lite-casual boxing fan, so he wasn't up-to-speed with some terminology).
I know full well that once its released the proper boxing fans will get in touch, and I'll change things post-release from that feedback, so my strategy is very much a post-launch end user feedback one. In an ideal world I would have had more friends who could help me test it, but there's only so many opinions I can factor in pre-launch, and managing their experiences and feedback would have been a mini-project to manage purely by itself. When already time-constrained with launch schedules, app store listings and last minute bug hunts and device optimisation I couldn't spare the time to manage a phalanx of testers.
This has been a passion project/scratch-my-own-itch project, which is why I feel more secure not having many outside opinions. If you're building something more strategic, to fill a target niche in a specific market say, then my approach will DEFINITELY NOT WORK. In this case you'd be best off firing out a message on LinkedIn or something like that, asking for would-be best testers.
That's currently where I'm at myself. Do you feel that, that approach scales fine and you get a broad enough idea of where problems are for the general user and where you need to improve/change things?
It doesn't scale, in that you ideally want to test your app's UI/UX with first time users who haven't gotten used to your app's quirks yet. But it allows to catch a lot of stuff early on at the cost of buying them a coffee or a beer (if that).
True, I'm not talking about UI/UX test, but tests that test that your application is "correct" and does the right thing. For such tests, you need to start out a beta or just wait for user feedback or also be great in product design and thinking like the average user. Adding help popups, and good documentations.
I have an app in the windows store with a few thousand users. When anyone emails me with feedback or an issue, I usually ask if they want to try the beta version (after I've resolved their issue of course). I've gotten about 15-20 part time testers this way, they've been a mixed bag. Some provide pages if detailed feedback, some barely anything.
When I have a decent, bug free mvp of the next version I'm going to fork out for usertesting.com
That actually sounds like a great approach! I can imagine you've triggered a sense of involvement or idea ownership in the most active ones. From my own experience, as a user, reporting a bug er suggesting a feature in a product, only to see it fixed or added, has a tremendous effect in loyalty and even promotion of said product. I think I'll definitely adopt this idea once properly released.
Forget usertesting.com use Mechanical Turk(get testers to download screen record plug in for chrome to record/send you their tests..charge $10 or less a test). Way cheaper and you get more screen recorded tests for less.
I do something similar. I have a Android app that I sell outside of the app store. I did build some licensing and support infrastructure to allow for automatic notification of updates. There is also a Beta release channel. Therefore, I use the support conversation in order to see if they want to be flipped to that channel.
I also use ACRA and Tracepot to capture crashes and anomalous events. Users can also email me a debug log along with their current app settings. This has evolved over a 5 or 6 year period.
I wouldn't ever pay for an agency to do user testing, and it has nothing to do with money.
Your ideal tester is someone who cares enough about your product to use it.
I also wouldn't even start developing the product unless I had a decent sized list of people who really want it.
In which case, testing it is easy. Just ask those people to test drive your app during development. If your app solves a real problem they will happily do it for free since it solves their problem, but you could also give them perks like giving them 2 months of your service for free when it launches.
Your users will naturally give you feedback if things are broken, so that handles post-launch real world testing. Of course you'll want to supplement that with heaps of automated tests too.
An agency or proper usability testing should provide you insights and ways to correct the usability issues based on the results.
What you describe may work to get feedbacks and new feature requests. But you won't know why a user is not using a feature, where they are struggling, what they don't understand or see.
> An agency or proper usability testing should provide you insights and ways to correct the usability issues based on the results. What you describe may work to get feedbacks and new feature requests. But you won't know why a user is not using a feature, where they are struggling, what they don't understand or see.
If you use heatmaps and other analytical tracking tools you can see how users interact with your app.
That's all you really need to see why a user isn't using a feature, struggling with the UI or happen to not see something.
An agency would just have X number of paid employees systematically going through your UI and giving you their opinions on what they think is best, but that might not align with what your real users are doing.
1. Stick the mailing list sign-up form on a landing page and do a bit of promoting in relevant places. Specifically say it's for beta announcements, preview release testing and picking everyone's brain. This should net you some users to bootstrap the process with, and they will be responsive.
2. Release the beta, mark it as such and ask people for feedback after they used the product for a bit. Listen to the feedback, act on what's relevant, keep improving the beta. Give kudos where they are due.
Have public forum, actively participate in all threads. Many people look at how active the support forum is before trying the goods or leaving any feedback. So the more active it is, the better. Also, moderate it aggressively to trim junk and trivialities.
3. Once out of the beta, don't forget to genereously reward everyone who chipped in. Ideally with a special gift, like a specially tagged lifetime premium account, a handful of most expensive licenses, etc. The worst thing to do is to offer a discount on a production version - this comes across as a cheap and greedy move.
I generally find sellers on fiverr. They are willing to record screen and comment for about 30 mins for $5. Not a bad deal and mostly good insightful feedback.
I do a free beta, then various paid betas. The goal is also to find a price point. A user can either leave feedback in my betas and get that month free, or they can pay. The idea that all they have to do is give feedback for a free service usually works well...
Eventually, when it's easier to pay, some users will start leaving money instead if feedback. At some point I'll raise prices a bit and stop the feedback for service system.
And a few others, it's always fairly successful - assuming you have a decent landing page and you can advertise the site in your target market for cheap / free.
Solo dev for an app with hundreds of thousands of users. These are the things I do that ensure smooth releases:
• run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users.
• fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.
• learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too.
• used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.
Probably a mobile app, and probably deployed via the platform tools. They have options for beta releases which users can opt-in themselves, and also phased releases are supported out of the box. Just my guess.
Also a "Solo dev for an app with hundreds of thousands of users"
I just test everything myself. On one OS (Mac). The worst part is having to run VMWare with older OS versions sometimes, but not usually.
I find that bugs are almost never OS specific, and are rather just logic errors.
I find that having other people test my code adds little value, as I can literally just test it myself in the OS that I write it in most the time.
I know that this goes against all advice that you read, and all the "common knowledge" that is out there... but I've been doing this for 15 years and have got over 1 million users in total probably, and I almost never get support emails regarding bug reports. So, in my experience, I really find that everything I have heard is wrong. I realize that other people may have other experiences but found this point of reference interesting.
Experience building software can lower the testing requirements.
I find the degree of testing needed can be relative to the difficulty of the problem, the unnecessary complexity of the solution, and the average skill of the developers working on all 3.
Clever (simple) architecture will often beat clever programming in the long run.
"This is totally going to work" is only said when I am completely sure that it is going to fail catastrophically. If anything works I feel mild distrust with my code from that point forward.
Knowing the difference between thinking less about yourself vs less of yourself is important when taking your work seriously but not taking yourself too seriously.
I have a healthy doubt of my own work that seems to end up making it work well first by keeping it as simple as possible.
Keeping the mindset of an innocent beginner is critical as a developer with experience. Everything is easy in the beginning, not the long term - I like to see what code bases people have worked on for many years instead of being able to neatly start from scratch.
Yep. I try not to get caught up in worrying by adding lots of tests and making my code easy to change. When I find an old crusty piece of code I actually have the most fun because it takes me completely immersing myself in it to find all the crap floating around.
Absolutely. It doesn't mean testing isn't required - it's absolutely the case when something is business critical, or average / inexperienced devs are painting unicorns and rainbows everywhere.
I have a healthy skepticism of my own work, it keeps hubris in check. This combined with a preference to keep things simple (to come back to later and to let others join easily), I find encourages everyone to build a code base they want to work on long term.
> fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.
Also check out Cent (https://beta.cent.co) - $20 in Ether can get you a pretty good set of responses. I am a cofounder - feel free to ama here or at cameron@cent.co
The way Cent works is instead of having 100 people earn 20¢ (for a $20 bounty), $20 gets distributed to the top 10 responses. Respondents are encouraged to be thoughtful because the community votes on which 10 responses get the bounty.
Many responses don't get paid as a result but making it competitive adds more value for the bounty-asker and makes the whole thing more fun in general.
Great gain for the bounty-asker, great loss for everyone > 10. Big win-lose gap. Just plain exploitation of the workforce. I don't see the fun in that.
The most important way to do this is to actually be a user of your product, e.g. "Eating your own dog food".
This doesn't mean logging in, trying all the features and playing with it for an hour - you have to really use it with your data on a daily basis over a long period of time. You'll come across a lot of bugs and start logging feature requests yourself.
Once you can use it without issues [and you'll need to be careful to split the wishlist features you come up with, against the actual bugs which need to be fixed], then you can get others to try it.
Best way to get user testing from others?
- email your users and ask them if they have had any issues, or want any new features.
- ask people to give your product a quick test - for example, https://www.lifepim.com is my new webapp to easily manage your personal information [shamless plug/warning - notes and tasks work well, contacts needs work, and calendar is not pretty]
- pay particular attention to requests from users who pay for your product. Everyone wants software to work in a certain way, but when someone pays, it can be a good indicator to listen to what they want. If they say something isn't working, make it a priority to investigate and address this.
Just keep in mind with dogfooding that there is a difference between figuring out how things work and working efficiently/effectively once you have figured out how things work. Dogfooding is really only good for 'testing' the latter, because you already know how everything works better than any normal user would.
The best part I've found with dogfooding is that even though you know exactly how it all works (and yes, you can miss some issues because of this) , actually using the product daily can make you realise - this isn't working well or that feature is kind of a pain in the neck.
Try to make a simple tutorial video of some sort. Even if it isn't useful in a practical sense it forces yourself (or someone else) to walk through the UI stating their intentions and trying to execute them. Some of the best feedback I've gotten was through watching videos like this for an application that I work on as you can see where the stumbling points are throughout the process.
Disclaimer: I've also worked on couple of those companies as a tester when I needed money and I was out of job, but thankfully I'm currently back working as developer again. On some of them, the tester also needed to record themselves using webcam.
Using your product daily is essential not only to help steer the ship on what to build but also to cue you in on what's buggy / inefficient. I have the good fortune of having a live chat interface (product is: https://www.jqbx.fm) so I get pretty quick feedback if something that's been pushed is causing anyone any issues. However to keep bugs to a minimum I keep most of the logic client side and have a dev environment that I can run on production- I usually test a version for a couple days (see: eat your own dogfood) before releasing anything. Also forcing yourself to review your own code is a good skill and has helped catch some issues right before pulling the trigger.
One thing I have found to be very useful is to ask my existing userbase of other projects if they want to test something new and give me feedback.
In fact this worked so well that I am currently building a new project around this. A site that lets startups gather feedback from people who are interested in trying out new sites and giving feedback.
It is currently in private beta with about 20 startups. If you want an invite, feel free to send me an email.
It doesn't matter how big you are, the same cheep method works for informal but fairly accurate quick and dirty testing.
Beer.
You think I'm kidding but if you go to a bar, there are lots of people who will test whatever you put in front of them for a free drink. If you want to keep your costs down even further then go during happy hour.
Is that "user testing" no... not in the sense we think of it today. Is it testing of the usability of your product: yes it is. Im guessing that most apps issues that are more likely the latter and not the former. Basic things that you think are clear that really aren't (your too close to the problem and the solution). If you explain, point and guide through the same aspect of your feature 10 times in 10 tests people might not get it! These are the problems you might want to solve, dumb it down, change it up, make it better.
Does this work? Yes, as long as you can moderate. If you can you be neutral and friendly and avoid "traps" that will alter your results. Saying things like "MY app" or reacting personally will alter your results, and learning when to "guide them" through a frustrating point and when to let them puzzle through it is a bit of an art (this too offers clues if your guiding people at the same place). For me personally a drink or two helps ME in the process...
I have also discovered that this process works very well with pairs of people. Two friends already have a rapport and will happily talk to each other to try to get through something problematic. One can encourage or discourage the other and their dialog can be enlightening. Letting them play together with your app and drink free drinks can give you a LOT more insight than the next three individuals testing. Anything more than two people becomes problematic so look to avoid large groups.
Guerilla testing is a great starting point. Do you usualy give one or multiple scenarios to complete?
Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)
I'm not OP, but I've used similar styles of pub/coffee shop testing. When I do it, I don't give scenarios or bother measuring things like success rate. It's very freeform, unscripted and raw. It's kind of like hallway testing for people who work alone. And, instead of coworkers, you're working with complete strangers who might not have any idea what's happening.
If I had to pick one thing to measure, I'd be more inclined to measure positive versus negative fucks (I'm not joking), but honestly, it's more about the experience and the totally raw, sometimes terrifyingly honest feedback I hear. Numbers aren't useful because the most useful sessions go way off the rails.
> Do you usualy give one or multiple scenarios to complete?
I am OP, and it depends.
Im old enough that I have done this with paper prototypes, and it is shockingly effective.
Now most of the code I write is compartmentalized enough that with a bit of effort I can come up with a one off digital artifact. Just rip out the chunk of your app that you want to test and present ONLY that. Spend a day building ugly shims early enough in the process and you might make some dramatic changes!
> Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)
Even with those things going on your going to be shocked at not only what issues come up but how frequently. That thing you did that you thought was simple and sane, no one gets it. Do more than half of your test subjects comment on the same thing (good, bad, both) then you have a problem.
The key isn't controlling the key is listening...
I also want to state that a slighty drunk, distracted bar patron, getting through your process without quitting, telling you your product is shit, or giving up is a positive sign that in the real world people who are more motivated than "beer" are going to have a fairly positive and expedient experience.
This is a great idea to gather problems/questions. There's no substitute for scaled A/B testing, and finding worthwhile things for testing can be the hardest part.
86 comments
[ 2.6 ms ] story [ 157 ms ] threadWhat's your goal? Do you want to test whether your update works correctly on many devices? Do you want to get ideas for new features? How many users do you have?
I have about 10,000 active users. I don't do much testing. I test myself on an Android device and an iPhone. Since my app is built with Ionic and based on Cordova, it works anyways and real crashes are not to expected.
Feedback is easy: I just wait until people write me an email. It's funny: Sometimes, you don't hear about a feature request or some issue for months. Then 2-3 people write you within a day. Then nothing again.
Back when I started, I implemented a lot of features. Now I basically reject almost every feature request, unless it comes up extremely often in emails. I sometimes take notes, but most of the time, I type a quick reply, thanking for the suggestion, archive the email and only try to get a feeling for what people want.
Obviously, this has its limits, but since I know my core users and what they want, it works for me.
If there's a serious bug, people will let you know soon enough.
I was thinking about just opening up on the app store as well, and just sit back and wait for reviews or emails, but that seems risky, since potentially bad reviews are forever. For a paid app at least, that can have a pretty serious impact on possible uptake. Having at least some level on confidence, that the app is usable and stable before general availability, is the safe play I think. To that affect, closed testing seems the only viable option.
You need to be make it as easy as possible to run your user test, because a tester will give up on roadblocks that take more than 60 seconds to solve.
I was developing a language learning product, and so needed testers who were interested in language X. In the end, the people I found using the filter [Interested in X language, native English] weren't actually interested in language X. Rather, they came across as people who had accepted the filter to make an extra buck.
However, it was good enough for a random internet trial. If I wanted better results I had to do it in person (which I also recommend).
There are no better user testers than your users, and the risk is only moderate. I could always pull the server down and restore the previous version while trying to fix whatever I broke in the deployment.
But if you aren't doing SaaS, user testing as a solo developer is not really feasible. The best user testing I can think of that doesn't "cost" is a Beta/RC Program for your users to volunteer to get potentially broke software for a discount or just early feature releases. For this to work though, you have to have enough users.
That's why I have always used a 3rd party system to collect actual usage analytics combined with server logs.
The above is what you get from usertesting.com and other sites but pay up the nose($50 a test) vs. $10 or less dollars per test.
If it’s a downloadable app Just spin up a web UI/UX and have the turkers follow same steps but have them change the browser to say iPhone 6/7/8 view.
In the next couple of weeks I'm due to release a boxing game (shameless plug: leatherthegame.com), and other than myself my only tester has been my cousin.
Having worked on the project for over 3 years I'm aware that there are areas where I "can't see the wood for the trees", but I am confident in my own knowledge, and only wanted an external viewpoint to see how a noob would see things from afresh (both in terms of not having used the app before and also him being only a lite-casual boxing fan, so he wasn't up-to-speed with some terminology).
I know full well that once its released the proper boxing fans will get in touch, and I'll change things post-release from that feedback, so my strategy is very much a post-launch end user feedback one. In an ideal world I would have had more friends who could help me test it, but there's only so many opinions I can factor in pre-launch, and managing their experiences and feedback would have been a mini-project to manage purely by itself. When already time-constrained with launch schedules, app store listings and last minute bug hunts and device optimisation I couldn't spare the time to manage a phalanx of testers.
This has been a passion project/scratch-my-own-itch project, which is why I feel more secure not having many outside opinions. If you're building something more strategic, to fill a target niche in a specific market say, then my approach will DEFINITELY NOT WORK. In this case you'd be best off firing out a message on LinkedIn or something like that, asking for would-be best testers.
I start out with manual tests.
If I have a bug at any part of the code I add a unit test.
When it gets too big and starts taking too long then I start writing e2e tests.
Its true that unit-tests ensure quality software. But this quality is a waste if real user's can't figure it out.
When I have a decent, bug free mvp of the next version I'm going to fork out for usertesting.com
I also use ACRA and Tracepot to capture crashes and anomalous events. Users can also email me a debug log along with their current app settings. This has evolved over a 5 or 6 year period.
Your ideal tester is someone who cares enough about your product to use it.
I also wouldn't even start developing the product unless I had a decent sized list of people who really want it.
In which case, testing it is easy. Just ask those people to test drive your app during development. If your app solves a real problem they will happily do it for free since it solves their problem, but you could also give them perks like giving them 2 months of your service for free when it launches.
Your users will naturally give you feedback if things are broken, so that handles post-launch real world testing. Of course you'll want to supplement that with heaps of automated tests too.
If you use heatmaps and other analytical tracking tools you can see how users interact with your app.
That's all you really need to see why a user isn't using a feature, struggling with the UI or happen to not see something.
An agency would just have X number of paid employees systematically going through your UI and giving you their opinions on what they think is best, but that might not align with what your real users are doing.
2. Release the beta, mark it as such and ask people for feedback after they used the product for a bit. Listen to the feedback, act on what's relevant, keep improving the beta. Give kudos where they are due.
Have public forum, actively participate in all threads. Many people look at how active the support forum is before trying the goods or leaving any feedback. So the more active it is, the better. Also, moderate it aggressively to trim junk and trivialities.
3. Once out of the beta, don't forget to genereously reward everyone who chipped in. Ideally with a special gift, like a specially tagged lifetime premium account, a handful of most expensive licenses, etc. The worst thing to do is to offer a discount on a production version - this comes across as a cheap and greedy move.
Eventually, when it's easier to pay, some users will start leaving money instead if feedback. At some point I'll raise prices a bit and stop the feedback for service system.
I've done this for:
https://projectpiglet.com/ and https://easy-a.net/
And a few others, it's always fairly successful - assuming you have a decent landing page and you can advertise the site in your target market for cheap / free.
• run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users.
• fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.
• learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too.
• used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.
Could you expound on this? How do you do that? (Related: How are you distributing? Through an app store?)
[1]: https://support.google.com/googleplay/android-developer/answ...
[2]: https://stackoverflow.com/questions/45073264/what-happens-wh...
What is your tech stack? What is your code-deploy process?
https://news.ycombinator.com/item?id=14542045
I just test everything myself. On one OS (Mac). The worst part is having to run VMWare with older OS versions sometimes, but not usually.
I find that bugs are almost never OS specific, and are rather just logic errors.
I find that having other people test my code adds little value, as I can literally just test it myself in the OS that I write it in most the time.
I know that this goes against all advice that you read, and all the "common knowledge" that is out there... but I've been doing this for 15 years and have got over 1 million users in total probably, and I almost never get support emails regarding bug reports. So, in my experience, I really find that everything I have heard is wrong. I realize that other people may have other experiences but found this point of reference interesting.
I find the degree of testing needed can be relative to the difficulty of the problem, the unnecessary complexity of the solution, and the average skill of the developers working on all 3.
Clever (simple) architecture will often beat clever programming in the long run.
This is a good point. The longer I've been at this, the less often, "this is likely to work" is a thought that I let go unchallenged.
I have a healthy doubt of my own work that seems to end up making it work well first by keeping it as simple as possible.
Keeping the mindset of an innocent beginner is critical as a developer with experience. Everything is easy in the beginning, not the long term - I like to see what code bases people have worked on for many years instead of being able to neatly start from scratch.
(joking)
I have a healthy skepticism of my own work, it keeps hubris in check. This combined with a preference to keep things simple (to come back to later and to let others join easily), I find encourages everyone to build a code base they want to work on long term.
Also check out Cent (https://beta.cent.co) - $20 in Ether can get you a pretty good set of responses. I am a cofounder - feel free to ama here or at cameron@cent.co
Many responses don't get paid as a result but making it competitive adds more value for the bounty-asker and makes the whole thing more fun in general.
What about this do you find so terrible?
This doesn't mean logging in, trying all the features and playing with it for an hour - you have to really use it with your data on a daily basis over a long period of time. You'll come across a lot of bugs and start logging feature requests yourself.
Once you can use it without issues [and you'll need to be careful to split the wishlist features you come up with, against the actual bugs which need to be fixed], then you can get others to try it.
Best way to get user testing from others?
- email your users and ask them if they have had any issues, or want any new features.
- ask people to give your product a quick test - for example, https://www.lifepim.com is my new webapp to easily manage your personal information [shamless plug/warning - notes and tasks work well, contacts needs work, and calendar is not pretty]
- pay particular attention to requests from users who pay for your product. Everyone wants software to work in a certain way, but when someone pays, it can be a good indicator to listen to what they want. If they say something isn't working, make it a priority to investigate and address this.
In fact this worked so well that I am currently building a new project around this. A site that lets startups gather feedback from people who are interested in trying out new sites and giving feedback.
It is currently in private beta with about 20 startups. If you want an invite, feel free to send me an email.
I know this isn't the answer most people want to hear, but no one tells the truth like live users.
Beer.
You think I'm kidding but if you go to a bar, there are lots of people who will test whatever you put in front of them for a free drink. If you want to keep your costs down even further then go during happy hour.
Is that "user testing" no... not in the sense we think of it today. Is it testing of the usability of your product: yes it is. Im guessing that most apps issues that are more likely the latter and not the former. Basic things that you think are clear that really aren't (your too close to the problem and the solution). If you explain, point and guide through the same aspect of your feature 10 times in 10 tests people might not get it! These are the problems you might want to solve, dumb it down, change it up, make it better.
Does this work? Yes, as long as you can moderate. If you can you be neutral and friendly and avoid "traps" that will alter your results. Saying things like "MY app" or reacting personally will alter your results, and learning when to "guide them" through a frustrating point and when to let them puzzle through it is a bit of an art (this too offers clues if your guiding people at the same place). For me personally a drink or two helps ME in the process...
I have also discovered that this process works very well with pairs of people. Two friends already have a rapport and will happily talk to each other to try to get through something problematic. One can encourage or discourage the other and their dialog can be enlightening. Letting them play together with your app and drink free drinks can give you a LOT more insight than the next three individuals testing. Anything more than two people becomes problematic so look to avoid large groups.
So probably how they will be when actually using your app? I do agree though in a reproduce-ability/data-integrity standpoint.
If I had to pick one thing to measure, I'd be more inclined to measure positive versus negative fucks (I'm not joking), but honestly, it's more about the experience and the totally raw, sometimes terrifyingly honest feedback I hear. Numbers aren't useful because the most useful sessions go way off the rails.
I am OP, and it depends.
Im old enough that I have done this with paper prototypes, and it is shockingly effective.
Now most of the code I write is compartmentalized enough that with a bit of effort I can come up with a one off digital artifact. Just rip out the chunk of your app that you want to test and present ONLY that. Spend a day building ugly shims early enough in the process and you might make some dramatic changes!
> Be aware that you are not controlling the environment and the users may be distracted by outside factors. Which may affect some of your metrics (time to perform tasks, success rate...)
Even with those things going on your going to be shocked at not only what issues come up but how frequently. That thing you did that you thought was simple and sane, no one gets it. Do more than half of your test subjects comment on the same thing (good, bad, both) then you have a problem.
The key isn't controlling the key is listening...
I also want to state that a slighty drunk, distracted bar patron, getting through your process without quitting, telling you your product is shit, or giving up is a positive sign that in the real world people who are more motivated than "beer" are going to have a fairly positive and expedient experience.
Doing this step first is a good way to avoid some of the worst case issues.
https://www.drunkusertesting.com/
good price, cheap compared to a dev’s salary, or compared to my time, paying less than $500 per month.
do 5-second tests, usability, card sorting, path testing, all ux and ui testing for greater chance of success