Last year I found myself in a rather unenviable situation: with only one week left to run usability tests for an online poetry magazine, I was experiencing incredible difficulty locating test participants who would be willing to spend as little as 30 minutes with me. The holidays were fast approaching. And although I wasn’t a fan of remote testing at the time, it became obvious I had to bite the bullet. Little did I know what a compelling option it would turn out to be.In-person usability testing is the most frequently used method of product research today, hailed as “essential” since as early as 1993. Back then Jakob Neilsen explained that “testing with real users is the most fundamental usability method …it is in some sense irreplaceable, since it provides direct information about how people use computers and what their exact problems are with the interface being tested.” Yet it’s precisely due to in-person usability testing’s prevalence that many people overlook the possibility of conducting their research remotely, in what are arguably more realistic usage contexts. In my situation, testing an online poetry journal with readers located in Australia, the US and the UK, it proved essential to include remote research. Not only was it impossible for me to travel to each location and recruit participants, it was also important to learn about — and see — each user’s behavior in their natural environment. For researchers learning about location-specific use, many will find, as I did, that remote research can prove more insightful (and therefore more effective) than its in-person counterpart.
What even is…?Remote research is any research in which participants and researchers do not interact in-person, face-to-face. It is typically conducted via a computer or a phone, allowing the researcher to see or hear (with the aid of screen-sharing software, for example) exactly that which the remote research participant sees or hears. This makes remote research a powerful tool for studying user behavior patterns “in the wild.” In order to gain contextual information, researchers often approach interview participants while they’re already performing a task on the actual website being tested, via a live chat feature or similar tools. This subset of remote research is known as time-aware research. It’s especially important with regards to mobile devices, where context factors heavily into a user’s experience. During the last five years there has been a significant increase in the tools by which remote research might be conducted. Nonetheless, remote research comprises only 5% of all user research. While A/B testing and surveys are both generally accepted as types of research that can and should be conducted remotely, usability testing is rarely seen as such. In my case, gathering real-time feedback on the usability of an online poetry magazine, remote participants were more focused on completing the task at hand than in-person participants, in part because they were not distracted by my presence. They were also more honest in their feedback, offering both negative and positive comments without any prompts on my part. Face-to-face participants, by way of contrast, were consistently verbally positive about the look and feel of the website, despite the fact that I watched them struggle to accomplish the tasks I had assigned them.
When to incorporate remote researchDespite its inherent advantages, remote usability testing isn’t right for every situation; there’s a time and a place for everything. In order to determine the best research methodology for a particular study, designers should evaluate their project against three key criteria: their research objective, their audience, and their budget.
Research objectiveThe first, and most important factor in deciding how to approach a study is determining the research objective. A researcher’s methodology should always depend on her objective, not the other way around. For example, if the research objective is to “obtain contextual information about the task at hand and its likely uses,” she will likely want to observe users in their natural environment. Remote usability testing, remote-access surveys, or remote A/B testing will all allow the researcher to do variations on this kind of fieldwork.
AudienceThe second factor to keep in mind when choosing a research methodology is the study’s target audience. Testing remotely with students and young professionals works particularly well, since this target group is generally comfortable with online meetings and screen sharing. This is important, because in some situations the participant has to be an active participant in accepting screen recording, turning their microphone on or even circumventing firewall restrictions. If the task at hand involves senior citizens however, who tend to have lower computer literacy, it can become complicated to set up remote testing—and technical glitches always occur. Or, if the target audience (regardless of age) uses vastly different terminology for their computer programs and websites, remote communication can become unnecessarily complicated. If, however, the audience is using a program similar to one they use daily, or if the program is intended to be particularly intuitive for first-time users, then remote testing can provide a far more accurate look at how users will approach the application. Essentially, considering our audience requires that we answer two questions:
- Will technology overly complicate the process?
- How can we best replicate our audience’s real-life use circumstances?