User Testing is Good, Live User Testing is Next Level
by Ethan Garr
Watching a user struggle to activate our product was humiliating.
But as a head of growth, it also taught me one of the most valuable lessons I ever learned.
User testing is great, but user testing with real people in the same room as you can be transformational. There are things you will only learn when you see them in person, and I learned this the hard way in a Starbucks several years ago:
It started well. My colleague and I explained the scenario to our tester: “You are getting daily harassing phone calls and you want to know who is behind the blocked calls. Show us how you go about solving this problem.”
Then we shut our mouths and watched. We were stoked when she Googled for a solution to the problem we described and quickly found TrapCall (our solution) and navigated to the site.
When she got there, she missed or ignored our “app download” icon, and began onboarding directly on the mobile website itself. We viewed ourselves as an app-first product, but right away we were learning that users might not have the same view. We thought we were on a roll . . .
But 15 minutes later we sat horrified as we watched our tester frustrated and stuck in a broken activation loop. She was practically rage-clicking, as our error handling and messaging just kept making things worse.
Eventually, she contacted customer support, but we knew most users would have just given up at that point. Then it got even worse as our CS agent got confused thinking she was onboarding on the app and could do nothing more than tell her he would call her back.
The whole thing was humbling. We had done user testing online, we had been looking at our analytics, how could we have missed what was clearly a common issue and one that could easily be solved?
There were lots of reasons for the failure. From limitations we faced in online user testing to an untested assumption about why mobile web conversions were always lower than other platforms, to flaws in our QA process, and so on.
But with live user testing, nothing gets masked. I know it won’t be possible for every product, but if you can find a way, run some of your user tests with real people right in front of you. You will find that these tests:
1. Highlight things that have fallen into your blind spots as a company
2. Fill in the gaps between traditional qualitative and quantitative analysis
3. Help your team truly see what customers experience with your product.
“The Mom Test” by Rob Fitzpatrick will help you do better user testing overall, and Daphne Tideman has a good post on how to get user testing going quickly. I will leave links in the comments below.
Hit me up here if you have questions about gaining valuable insights from your users.