User Testing #2: Selecting Respondents

Why to test, what is the testing process, and how to recruit respondents on an ongoing basis, we discussed in the first part about user testing. Today we will look at screening.

‍

‍

Screening means selection of respondents based on the necessary characteristics (demographics, knowledge, behavior, habits, attitudes, etc.) according to the specific needs of the project. You always need to test with the relevant target group. Yes, always. Sure, if you're designing a Brno website, you'd think it's enough to reach anyone in Brno... What about a woman who doesn't use a computer or smartphone and has a vague idea that there is an internet? I understand, that's an extreme example, but even that can happen -- we'll get to that. It is simply a matter of choosing the criteria by which you filter the candidates.

‍

How does it work?

First you need collect contacts. You can use your friends, acquaintances or family to recruit. Furthermore, social networks, community forums, blogs. You can also collect directly via the website or at branches, conferences or fairs. There are many possibilities — and as we pointed out last time — collect contacts continuously and whenever you can.

In the second step build queries projecting the criteria based on which you select testers. Write down who you definitely want and how you can recognize them (they play online games/trade on the stock/use a book reader/etc). At the same time, also be aware of who you want to avoid and what criteria define them.

Then to all those who are considered, send a questionnaire and with those who comply, agree on a testing deadline. You can also do the screening very quickly by phone. At the same time, you will verify the communicativeness of the respondent and their interest in participating in the research.

In the field, you can then capture existing customers with a questionnaire (for example, in a store). Suppose you want the childless, from the upper socio-economic class, who often shop online via smartphone. With the same requirements, you can of course also collect on the web. This way you reach a larger audience, but you have to deal with the necessary location. If you formulate the screening questions correctly, all you have to do is call the respondents, verify the screening criteria and arrange the testing. Also, do not be afraid to take advantage of the different groups on facebook where the target groups are concentrated.

‍

Never reveal everything

Regardless of the method of verifying the relevance of respondents (online/offline/by phone), one rule applies here. Don't tell me what it's all about. Reveal only what is absolutely necessary. It is difficult to hide the fact of what kind of company it is on the e-shop, but you do not have to reveal that it is user testing — just tell the research if the company does not want to reveal that it is developing a new website. If you're screening by phone or email, don't reveal the name of the project, people will want to meet you and click through the site ahead. Plus, you destroy the initial 5 second test (more on that later).

If the respondents do not meet the criteria, thank and end the interaction. If they suit, you can them ask for contacts with similar people from their social circle. Young moms will probably know other young moms, etc. It's called the snowball method.

‍

Demographics are a coarse sieve

How do you define who you need? Typically, you solve socio-demographic factors (sex, age, status, residence, education, employment, income, employment, etc. ), Knowledge of the brand, frequency of use of the product/service, behavior and strategy, experiences, use of the technology etc. There is no universal guide to screening questions -- it depends on who you need, who your target audience is, and what defines it. See above — write down who you want and how you know them and form appropriate questions.

‍

‍

Beware of the isolated use of demographic questions — they are not a reliable indicator of whether a user is right for you. It's a coarse sieve. Just because two people are thirty doesn't mean they are alike in anything else. Combine is therefore with questions that tell you more about the strategies, behaviors and habits of the respondent — if relevant to the target group.

‍

Don't forget the escape answer

Questions to the screener must be simple and understandable. Everyone needs to understand them equally and understand what you're asking them.

However, it can sometimes happen that respondents do not know, would not select an answer, or are not sure where to rank (which may indicate a poorly chosen screening question). However, for these purposes it is good to add the so-called escape response, i.e. Other, dunno chi None of the listed (according to context). Otherwise, the respondent may falls into our coveted category because he had no choice.

Let's take the following question as an example: Do you think you are information-literate?

1. The question is not understandable. Information Literacy is an academic term confused with computer literacy. No one knows much what that means, the respondent probably doesn't at all.
2. The question is closed. Respondents will gravitate toward answering yes because they feel like you want to hear it. Since they additionally don't know what they're replying to, it wouldn't tell you anything. And most importantly, you will get richer data if you offer answers and thus determine the rate, not just whether it is or not. However, even closed questions have their place in the screener, for example if you need to sort out the respondents quickly at the beginning.
3. The question ascertains beliefs, not facts. If you are wondering how well the respondent can use a computer, you can list different concepts and ask if he could explain them to his grandmother. I am referring to groups according to intensity, for example:

• e-mail, google.com, .cz
• Facebook, Skype
• RSS, podcast, blog
• CSS, FTP, WP
• jQuery, SOAP
• I don't understand any of the listed

‍

Thanks to this, you can categorize the respondents and most importantly weed out IT professionalsthat in the vast majority of cases you do not want to test with because they have a distorted view.

Alternatively, include typical activities on the Internet in the answers (the frequency of use can be deduced from whether a person watches TV shows and parties games, or watches TV programs and weather). Variants are innumerable.

‍

Not a basket like a basket

Finally, I would like to illustrate the importance of screening with a short story from user testing. You will feel the importance of screening painfully the moment you get tested by a respondent who sees a computer for the third time in his life. Yes, that's what happened to me.

Although the screening questionnaire at the store was filled out by a lady who shops on the Internet several times a year (which was our criterion), she forgot to mention that he She's filling in for her sister to make some extra money too. (respectively, she received a voucher for the purchase on the e-shop). In short, she misted.

How was the testing going? Insecurity, fear, a clenched hand that is afraid to ride with a mouse and does not know how to use it. At this point, I had two options. Either end the testing immediately and not waste time because the respondent is not the target group or slow down the pace and continue. Curiosity and fascination by observing someone who just can't get your hands on, unambiguously won.

‍

‍

I was kind, patient, but persistent. I have come to realize that the lady at all had no idea what the shopping cart icon meant and she would order the goods by finding a phone number and reporting what she would like to deliver. She thought that clicking on the cart icon to short display the cart menu (with which he can fill the cabinets that he also offers in the e-shop). It doesn't sound so illogical in the finale, does it? I click on the basket, I will see the baskets. I recall that a 35-year-old respondent said that she had heard that it was possible to shop online, but had never tried it.

However, the respondent's learning curve was fascinating, the first tasks took her an extremely long time, but once she understood the principle of how the e-shop works, she completed the scenarios relatively confidently and successfully.

Back to the point. Although the respondent underwent screening (filled out a questionnaire determining the frequency of purchases on the Internet, age, job position, whether she owns a smartphone, etc.), when arranging testing we had the answers reverified.

And in the end, what you're probably most interested in. How many souls for testing from one target group do you need? Ideally 5.

In the next part, we will look at how to build a test scenario.