by guest author, Cade Diehm, founder of New Design Congress
All design practice requires user testing. Regardless of the field or medium of design, user testing should be a core part of the design process. User testing ensures that your proposed solution can be understood and operated by people. It is critical in all stages of design, from prototype to release candidate. This chapter covers practices that can be used in-person or remotely for user testing, while understanding the opportunities and barriers of remote testing for your product or service.
In-person user testing is often done in controlled environments, where a design team selects sections of their current work to test or tests the entire experience. The design team would establish an on-site environment with team-controlled devices and observational equipment. The team then recruits testers, scheduling one-on-one or group appointments between test facilitators and a cohort of users. User testing is qualitative, in that it focuses on individual users and their experiences with what you are testing. This is true whether the testing is conducted as one-on-one sessions, or as participatory testing in larger groups or communities.
Remote user testing uses a similar process to its in-person equivalent, except conducted online. User feedback during on-site user testing is usually conducted verbally and observations of the user testing session is conducted and documented visually. In remote user testing, the entirety of the test is facilitated via digital platforms and technologies. Remote user testing differs significantly from its “local” counterpart, despite having the same objectives and, in many cases, producing similar data.
Note: User testing — whether local or remote — is not the same as bug reporting or beta testing. The purpose of both on-site and remote testing is not to identify and document software bugs, design flaws or hardware issues. Finding flaws in a product is handled by Quality Assurance, which focuses on discovering unexpected behaviors or issues in designs that are finalized and occur closer to launching the product or service.
User testing is the process of testing the ideas produced by a design team. Although the motivation of user testing remains fairly common across projects (to see if the design meets the expectation and comprehension of users), the goals of user testing can vary. Goals for user testing might be to:
Good user testing — especially testing as part of a participatory design, human-rights centered design, or an anti-weaponised design process — accomplishes its goals through collaboration between the design team and test participants. The design team may have specific goals in mind, but the main goal of the team is to present the testing and user walk-through of the design in an unstructured way, balancing direct instruction with exploration and input from the participants. User testing is extremely effective when the test design and facilitation process incorporates the participant and community in such a way it embraces the expertise of participants and their social environments.
Remote user testing is the practice of facilitating testing with users and communities where the facilitators and participants are not in the same physical space. Instead, the design team uses digital platforms and tools — video calling, remote access, digital whiteboards, live prototypes, etc — to facilitate the testing and research.
In situations where you are unable to physically access communities, such as the Pandemic, remote testing is often the only solution. Testing with users in vulnerable regions, where physically meeting with participants may put them at risk, is a similar situation where remote user testing may be required. Much like all ethnographic and/or social research, the safety of participants is paramount. Remote user testing, combined with strong ethical controls and security practices, makes the safe facilitation of user research more likely.
Remote testing is also appropriate in situations where the economic conditions of either the design team or the user base make it impractical to perform in-person testing. This may be due to the design team dispersed globally, where the logistics of travel make in-person testing impractical. Remote testing is also useful for under-resourced teams, such as projects staffed by volunteer designers.
Building multicultural user experience and testing against different audiences is another good use case for remote testing. Testing internationally, in which the design goal of localizing the design or cultivating multicultural user experience, can be difficult without the privilege of a large supporting budget. Remote testing may not solve the challenges of localization or communication between facilitators and participants, but offers low-barrier access to communities across the globe.
Remote testing can also be institutionalized as part of an organization or project’s participation with the community, offering regular structured feedback sessions with communities or participants who have a stake in the design team’s work. In many cases, this is incorporated into an “early adopters” program as part of a community engagement and outreach plan.
Remote user testing is often selected due to the circumstances of the design team. Despite this, the approach offers a wide variety of benefits that are common to almost all remote testing projects, regardless of motivation:
Although remote user testing has significant benefits, the process also has some overall potential drawbacks. The challenges of test logistics, the legality or ethics of using participant devices, testing complexity, and other issues can compromise remote user testing. Many (not all) of these can be mitigated via careful planning by the design team, as well as regular review of test structures and dedicated feedback rounds with participants.
The logistics of testing with an individual participant is often shared between that participant and the test facilitators. For participants with less technical expertise, this can be a significant or confusing burden. Working with a participant to set up a test environment, install test software, or gain access to the designs can result in communication, frustration, and stress issues for both the participant and facilitator. This is the antithesis of participatory design. Rather than meeting participants and communities where they are, remote user testing logistics often create experience silos where the individual participant is on their own with the facilitators and shouldering additional responsibility.
Additionally, the reliance on the user’s own devices to complete a test session can be variable. Unless a design team is particularly well resourced, it is often infeasible to distribute test devices remotely to participants or communities. As a result, remote user testing occurs mainly with devices the participants have on hand. This can have significant implications. Not only are there ethical considerations in the economic and/or power dynamic of a test participant and their devices accessing the design, but participants potentially open themselves up to intellectual property or copyright liability by having a copy of the test on their devices. The ethics of deploying test software on user devices is problematic without careful consideration or mitigation. There are issues of code stability, digital safety, and other concerns for the participant. Often, participants are unfamiliar with the design and its broader social dynamics and goals, and as such are unaware of the risks they are consenting to. Depending on the sensitivity of the project, these test designs can be used to target the participant simply by participating online in a way that can be observed by a third party.
Testing on participant hardware often allows facilitators access to niche hardware that is used by the individuals for great accessibility online. However, the lack of familiarity of this hardware from the facilitating design team carries substantial risk. These tests can encounter unexpected obstacles, where tools for observing participants interfere with their accessibility hardware, or render either the test or the participant environment completely inaccessible.
Depending on the project, remote user testing can benefit by being closer to a real world test case. But the nature of today’s personal and mobile computing prioritizes a “1 user, 1 device” paradigm. This means that, in many cases, the remote test itself is severely limited, as the test is filtered through platforms that adopt the 1:1 user/device design model. For participatory design, this is particularly tricky, as performing remote user testing simultaneously with inter-community participants is often overwhelming and difficult without careful test design.
While drawbacks are inevitable in remote testing, having good feedback loops between participants and facilitators where they are able to properly evaluate the tools and platforms that may be used as part of the test prior to testing can help minimize setbacks.
User research is conducted on people and communities and is subject to research ethics. For in-person testing, many of the human research ethics questions have been extensively researched in the adjacent areas of social science and philosophy. For remote user testing, research ethics are more complicated and existing recommendations are less useful.
Beyond the expectations set out in existing ethical frameworks for researching with humans, the ethical implications of technologies, networks and data remain vague and under-explored. For example, most testing tools are built by Silicon Valley companies that have economic relationships with data brokers, who will sell the data associated with your testing to their customers. In testing plans that include multiple platforms, this multiplies the number of privacy violations exponentially. A design team may think that they have nothing to hide, and not worry about the issues related to platform surveillance. Other teams may choose to only use platforms that allow them greater control. In all cases, the design team is making a judgment call on the safety of their participants. Their decisions may have profound and unforeseen consequences for their participants, especially as societies change, companies change leadership or even a participant’s individual circumstances change over time. A good general rule is to collect as little data as needed, and nothing more. This can be applied to both the research data generated and archived by the facilitators, but also the data generated for the platform operator by the testing session.
Remote user testing can also require test participants to install intrusive software on their computers. These software tools include remote control software (like TeamViewer or similar applications), proxy software (to enable a participant to access confidential parts of a test), and other surveillance tools. Almost all of these tools have the potential for abuse, and their use can help normalize the opportunities for abuse. For example, TeamViewer is a remote screen sharing and desktop control tool that can streamline testing sessions, but also an iconic example of weaponised design. Its aesthetics, feature set and interface makes it a staple software application for technical support phone scams, where fraudsters deploy TeamViewer on a victim’s computer as part of a sophisticated attack to control their computer and empty their bank accounts. Design teams should think carefully about the software they use and how they communicate the harm to participants, as well as instructions for proper removal after the testing.
Remote user testing also has a significant power imbalance between participant and facilitator. The inability for either side to see each other fully, combined with the observation of the participant by facilitators creates a hierarchy of visibility that can have serious ethical consequences. For example, participants are in many cases unable to see what the facilitators are doing, and must trust the facilitators to communicate when they start or stop the session recording. With in person testing, there is often a stronger implied social contract and the physical presence of facilitators and testers helps to lower this power imbalance. Other issues also contribute to this power dynamic, such as data governance and privacy of the test data produced during a session.
Participants are also sometimes asked to run the code of the design being tested, either on their computer or within their web browser. This means, in many cases, the design you are testing will leave the custody of the design team and be copied onto the participant’s device. This creates an additional concern for the participant, who in many cases has to commit to removing the test ephemera (and know how to) once the test is concluded. Teams should think carefully about asking participants to commit to this, and explore the potential risks associated with making this a component of their remote user test.
During in-person testing, the test facilitators are able to deploy and completely control the testing environment. Device administration, observation equipment, and other tools are all expected to be supplied by the facilitators, or in cases where participants bring their own devices, the on-site support from facilitators during the test is invaluable. The opposite is true for remote testing, and this creates perhaps one of the most common and significant challenges encountered during remote testing. Setting up the test environment is a collaborative process between the participants and the facilitators. This takes time and can be fatiguing to the participant. In the worst case scenario, a complex setup process may be beyond the abilities of the participant, and prevent the test from happening. When planning remote user testing, facilitators should minimize the setup obligations they place on the participant, and provide ‘off-ramps’, or alternative means to complete the testing session in cases where the participant encounters obstacles at the beginning of the test.
Hardware, software, and connectivity requirements have an under-appreciated but significant impact on the viability and accessibility of remote user testing. As computing power has increased over time, and the reach of wireless connectivity expands, software platforms too have expanded to fill these newly available resources. Remote user testing can require a specific set of applications running simultaneously. When combined with intensive tasks such as screen broadcasting or group calling, or the need for constant connectivity from participants’ connections, can create technical limitations for participants. It is naive to assume that this problem is solely driven by economic factors — many popular Apple laptops from the last five years are incapable of running many resource intensive tasks at the same time without encountering significant performance issues. These technical accessibility concerns have a direct impact on a remote testing session. Teams can prevent this by refraining from running multiple resource-intensive platforms simultaneously, or offering flexible testing pathways for participants.
Remote testing platforms or the act of using a platform familiar to a participant in an unfamiliar way can also affect the accessibility of the test. The dominant paradigm of user experience design, Don’t Make Me Think, has created a collection of interaction paradigms that emphasize simple, inflexible means of operating software. When these are combined in unusual ways, or when moving between two incompatible platforms during the testing process, the changes in user expectation can be jarring. This can have a significant cognitive cost for the participant, and is something that can be supported with ease during in-person testing. Teams can prevent this accessibility problem by exercising restraint in their choice of platforms, and by setting exploratory expectations with participants at the beginning of the test session.
In cases where a design team is testing with an inter-sectional sample of participants with diverse accessibility requirements, the testing environment can conflict with accessibility software and hardware used by the participant. For example, a test environment may not follow accessibility legislation, and as a result the participant’s screen reader cannot render the platform’s interface in a comprehensible way. Or, issues with the testing platform could act as an interference layer, preventing the participant from interacting with and evaluating the design, causing the test to fail. Mitigating conflicts between testing platforms, design and accessibility tooling requires careful planning and preparatory research from the design team. In cases where the team has advanced awareness of test recruits with accessibility peripherals, scheduling short platform tests or contacting the developers of a proposed test platform is a starting point for preventing unrecoverable accessibility obstacles during testing.
Testing A Tool For Recording Interviews
In a recent project, my design team sought to test an experimental user experience tool for navigating and exploring recorded interviews that was designed during the pandemic. The design team wished to test the tool’s design, differences experienced across cultures, and the relationship between spoken word recordings and text.
The team structured the remote user test as a shared participatory workshop activity. Test participants were selected via a call for participation online, where both the workshop activities and the goals of user testing were explicitly detailed. Participants were selected from the broader international network available to the design team. The team identified a single design they wished to test, and structured the entire remote-testing process around this design. Participants signed up for one of two workshop choices, structured to cover different time zones.
The remote user testing sessions were held over a group Zoom call. After an introductory icebreaker, the two test facilitators detailed the consent and ethical commitments from the team, and other introductory processes. The design team’s two facilitators described the design scenario, and the participants collectively operated the tool, giving verbal feedback and collective reflections in real time. The test facilitators recorded this feedback. Each test session / workshop took 1.5 hours to complete.
At the conclusion, participants were invited to reflect collectively on the test process. A few days after the test sessions, the design team contacted participants individually for written feedback. Facilitator notes were combined with summaries of the recordings of the test sessions, and presented back to the design team.
User testing is a core design practice and is used to uncover everything from unrealized socio-technical security threats to cognition difficulties in a designed system. Remote user testing, the practice of performing user tests over a network, allows designers to reach a wider range of people and communities to collaborate with, but not without significant challenges inherited from limitations of the internet and personal computing. Careful planning and sensitivity to issues of accessibility, colonialism, institutional power and safety can create the conditions for successful remote testing that can compare measurably to a design’s consequence in the world beyond the bounds of its creators.
Step 1: Break up the in-person test into smaller tests
Although in-person tests can be long and ambitious, remote user testing has a significantly higher cognitive load for both participants and facilitators. A remote-user test should be designed to take this into account, and test more frequently in shorter sessions. Decide how to break up your testing into different parts. To ensure the participants are fully heard, facilitators should plan for a post-test debrief, where the participant has more time to reflect on the test.
Step 2: Structure testing to be remote-friendly
Unlike in-person, remote testing has a more distanced dynamic between participants and testers. Consider the techniques of test instruction, participant observation and feedback, and adapt the structure to your remote testing environment. Facilitators can use participant webcams (with explicit permission!), plan participatory tests with breakout sessions to allow for smaller environments, or offer collaborative documents for feedback from participants to complete tests that yield valuable research data.
Remote user testing usually consists of a variety of platforms used together to facilitate participants in the process:
Remote testing may include some or all of these platforms. Their use is dependent on the requirements of the testing, and the institutional dynamics of the team, participants, and community. Write down which platforms you will use as part of your testing.
Step 3: Plan each step in the test
Remote user testing relies on additional logistical responsibilities for participants. Complete a test plan that should be meticulously designed to ensure participants can succeed in joining and completing the test. Evaluate the platforms you intend to use in your test, and remove as many complexities as possible. Account for edge case scenarios where problems can occur — such as a participant’s microphone or camera failing during the test — and develop test flows to cater for these situations. Design teams can explore extreme (but possible) edge cases by designing games that review different scenarios and mitigation strategies collaboratively.
Facilitators that are sensitive towards and plan around the cultural and cognitive diversity of participants and participating communities will produce high quality test result data. These teams will plan tests that account for Indigenous or diverging pedagogy, and de-colonise their methodologies to create space for participants to direct and involve themselves most appropriately in the test and its environment.
Step 4: Test the testing process
To decrease the risk of test failure, facilitators and test planners should test the designed mitigation strategies. Dry-running tests with simulated equipment failure, connectivity problems, and other obstacles will help facilitators to respond quickly and decisively. With proper mitigation practice, this process will increase the chance of recovering from a failure, allowing facilitators to complete the remote test with the participant.
The remote nature of testing requires facilitators to adopt a flexible but confident approach during the session. Supporting participants, particularly during moments of confusion or uncertainty, will help prevent the test from losing momentum and reduce the chance of participants failing to complete the test.
Step 5: Host the test with participants
When performing remote user testing, there are additional steps that facilitators need to complete. Once tests are scheduled with participants (usually through appointment software or other scheduling platforms), the design team should send a Participant Research Consent form to the participants well in advance of the test. This form should ask for consent, provide a research ethics statement, detail how participants can withdraw consent, and include additional information about privacy and data storage. Remote testing should never be held with participants who have not signed a consent form! Instead, facilitators should remind the participant before the test and reschedule rather than press the participant to consent.
As the test date approaches, facilitators should send participants a clear description of the test to set expectations, and provide clear, plain-language instructions for preparation and entering the testing environment. These instructions should detail all technical requirements, and provide advice for minimizing the unintended sharing of personal information (such as personal items in the background of the room being broadcast via webcam, or device notifications captured during screen-share.) These privacy preparations should be repeated to participants when they first join as a reminder.
At the time of the test, make sure participants can reach the facilitators via commonly used communication methods (e.g., email or instant message). Once the participant has joined, the facilitators should run through a pre-test introduction, review the privacy statement and consent form, and clearly state when the test is in process and recording has started or completed. Facilitators should also take time to allow participants to familiarize themselves with the testing environment and tools being used before the test begins. For example, explaining how a collaborative writing platform for feedback works, or explaining mute/unmute camera on/camera off on a screen sharing platform.
As much as possible, remote tests should be completed in a participatory way. If you’re testing a group, this can be done through breakout rooms offered through the video platform. When facilitating these kinds of tests, test facilitators should take great care in ensuring that all testers are able to keep pace with the testing group, and able to provide feedback on their own terms while not being biased by others' responses. Good, open moderation is key, to ensure the test represents the experience from all participants.
In this step prepare the description of the test and testing environment, along with the pre-test introduction and the privacy/consent statements.
Step 6: Capture, store and secure the test research data
When recording the testing process, facilitators gain the ability to document exactly what they see, rather than relying on interpretation or notes that may be generated from an in-person user test. This does have benefits, such as allowing facilitators to focus fully on test logistics and interactions with participants. But facilitators should pay great attention to the methods of capture and ensure they are reliable.
There is no set formula that can identify the “correct” ratio between test participants and facilitators, however tests should ideally be conducted by more than one facilitator in dedicated roles. If facilitators are recording the test, then multiple recordings should be taken simultaneously — either by the platform (if the platform offers a recording feature), or by each of the facilitators. Once recorded, the testing data should be copied as soon as possible from the platforms or facilitator computers, labeled, and transferred into a reliable storage location, along with the participant consent forms. Remember that, by conducting user testing, a design team likely has obligations under the European Union’s General Data Protection Regulation (GDPR), the Chinese Personal Information Protection Law (PIPL), California Consumer Privacy Act (CCPA) or other local data regulation laws. Identify the tools you will be using to capture the testing, how you will store it, and any additional security measures that need to be used.
Step 7: Follow-up with participants
Once complete, a follow-up with participants is essential to open the feedback loop and iterate on the testing process. This can take the form of a survey or informal interview. Understand where in the timeline you can follow-up with participants and the method you will use.
Beyond the feedback loop, following up with findings or design evolutions as a result of the test research data is a core component of a participatory design methodology. User testing in this way creates further opportunities for iteration or ideation workshops with participants, which increases their stake in the design. Good examples of this include the Dual Power Open Design & Build process, or the CCP’s player-led Council of Stellar Management in EVE Online.
Further Reading The Limits to Digital Consent