As a user experience consultant, I occasionally use remote online unmoderated usability testing tools. Two of the most popular tools that I've used are Loop11 and Usertesting.com. Part of their popularity is because they provide the option to recruit participants for you. If you've recruited participants for a usability test before, you know that it can be difficult and time-consuming to find and schedule qualified people, making these all-in-one services very attractive. However, if you examine what it takes to be a participant for Loop11 or Usertesting.com, you may begin to wonder how qualified these participants actually are. To find out what it is like to be a participant, I signed up to be a tester on Usertesting.com and on Loop11, which uses Cint panels in the United States. What follows is my experience and my opinion about these popular online usability testing tools.
My Usertesting.com Participant Experience
Over a few weeks, I spent around 3 hours answering screeners for Usertesting.com tests. I qualified for exactly 2 tests, and earned a whopping $3. I can think of many ways that my 3 hours could’ve been more wisely spent, so I’m not entirely sure who has the time and patience to be a participant.
As a participant, you are given a bevy of surveys on your dashboard, which are screeners to see if you qualify. An astounding number of these screeners ask if you work full-time, part-time, or are unemployed, and what industry you are in. Sometimes they ask for annual income. The screeners are redundant, and it seems like something that could be cleared up with a more robust participant profile. After one or two of these questions, you’re either qualified or not qualified. If you’re not qualified, you don’t get to take the test and you aren’t paid.
Once you finally qualify for a test, you might soon discover that what was supposed to be a 20-minute test is actually a million-minute test. I’m not kidding. I qualified for one test that had 26 total questions. As a UX professional, I know that this is impossible in just 20 minutes. I was only on question 6 at the 10-minute mark so I exited the study. The company wasn’t paying more than $10 for this test, so it was not worth it to me to continue, especially since this test seemed like it would take closer to an hour.
However, in order to be paid for a test, you have to complete it. If you rush, Usertesting.com’s customer can give you a bad rating or refuse to pay you. A low rating negatively impacts your ability to get more tests. So, as a participant, you have no recourse against customers who are wasting your time. There is no way to report or rate them, and no way to recover your lost time. Before you think this was just one bad test experience, a coworker of mine tried Usertesting.com for much longer than me, and reported having this same experience.
The other test I qualified for was well-designed, except the screener was generic and I was probably not the true target market. The language used on the website was British English, and they used quite a few different terms that were not common for an American English speaker. Perhaps this was what they wanted, or perhaps they should’ve screened people from the U.S. out. This was short test, designed to only take a few minutes, so my Paypal account was credited $3. Overall, it was a good experience.
Cint Participant Experience
The Cint participant experience was much better than Usertesting.com, primarily for user experience reasons. First, Cint has an extensive participant profile to fill out that is optional, but asks very typical market research questions. Thus, their screeners are not redundant and you are more closely matched to relevant opportunities than you are with Usertesting.com. Also, their workflow is very well designed - Cint keeps feeding you screeners, so it's rather addictive and easy to use.
However, I never actually encountered a Loop11 test via Cint in the 2 hours I spent taking screeners, and I only qualified for one survey. Because I've used Loop11 before to conduct usability tests, I know that Loop11's customer also has the ability to reject usability tests if they were rushed through or not thoughtfully completed, so you might take a test and not receive payment. And, though I didn't complete a usability test via Loop11 as a panel participant, the quality of the actual usability test and screener is dependent upon the skill of the customer who created it, just like with Usertesting.com.
Payment-wise, it seemed like all of the screeners I completed were promising between $1-1.50 for a completion. Based on the prices for running Loop11 usability studies, I know that it is probably not possible to make Usertesting.com-style money as a Loop11 participant. Somehow, the monetary value of my time bothered me much less as I completed the Cint screeners. Perhaps this was because the Cint payment amounts were much lower stakes than the $10 offered by Usertesting.com, or maybe it was due to Cint's interface design pushing me through dozens of screeners, making me feel productive.
Usertesting.com vs. Loop11: Which is Better for Recruiting?
As a usability professional, attempting to be a participant for both of these services gave me qualms about using their panels for recruitment. For many usability tests, I’m not certain you would truly get qualified participants with their panels. Years ago, I was at a conference session about using surveys for user research, and the speaker pointed out the big problem with validity of survey results - only a certain type of person bothers to complete them because they are a waste of time for the end user. You have to be willing to settle for feedback from anyone, whether they are the right customer for you or not. This is true for usability test participants from panels constructed of individuals who are specifically seeking tests to take for payment.
For this reason, capturing participants from your own website or recruiting from your social media channels and email lists is a better way to get feedback about your real audience's needs and usage patterns, even though it might require more legwork from the UX team. Loop11 offers visitor intercepts on your website and using your own participant lists as a recruiting option, but Usertesting.com only offers using your own participants in the pro version, which is not pay-as-you-go. If you want to use your own participants and don't have the resources to go pro with Usertesting.com, Loop11 is the better choice of the two.
Why use participant panels?
Of course, built-in participant panels are not all bad. First, you can absolutely uncover basic usability problems with the participant panels provided through Usertesting.com and Loop11's Cint relationship. For example, you can identify an image carousel that changes too quickly, or common web conventions like navigation and search that aren't behaving the way an average user expects. These are issues any tester may find, and this type of functional usability testing can resolve major usability impediments.
Another time when these built-in recruiting services do well is when everyone actually could be a participant. For example, e-commerce retailers with mass market appeal, health information websites like WebMD, news sites, recipe websites, etc. However, these are the easiest participants to recruit because they are also your neighbors and friends, and can even be intercepted at your local coffee shop.
In the end, for the most valuable usability testing results, you'll want to make sure your website matches your participant’s reality, and even their vocabulary. Therefore, if your website isn’t mass market, or if your issues are deeper than pure function, you will be better served by a more targeted participant recruiting strategy than a built-in participant panel.
Want to know more? Check out my other posts about usability testing methodologies and tools.