Relevance & Research Question

The quality of a sample remains one of the main problems in online research; special emphasis is put on the non-response problem and response propensities. Much effort has been devoted to determine the probabilities of being included in a sample and understand the differences between respondents and non-respondents.

In order to model the abovementioned probabilities, researchers can contact individuals chosen to the sample repeatedly. We can assume that users who reject first invitation are less approachable than those accepting it. Therefore, multiple attempts of contact may improve the sample quality and provide additional information for the weighting process.

To test the outcomes of such approach and compare characteristics of “first-contact-users” with “further-contact-users”, we conduct online survey with the use of improved intercept survey method (on-site recruitment) – as the issue of non-response is particularly troublesome in studies carried out this way.

Methods & Data

The questionnaires are displayed on over 60,000 websites embedded with research scripts allowing us to use information stored in cookies. Algorithms of simple random or stratified random sampling are applied to the cookies visiting these websites. Afterwards users who have been exposed to the pop-up invitation but did not take part in the study are shown the invitation again upon entering one of the mentioned websites after at least 12 hours from the first invitation.

Results

The results confirm that a sample collected thanks to successive invitations comprises respondents generally less willing to take part in online surveys than those from a sample collected with the first one. We can also observe differences in the demographic and behavioral profile.

Added Value

The approach shown in the presentation can be helpful in both obtaining less biased samples (by including less accessible users) and understanding the characteristics of groups underrepresented in online studies (by comparing characteristics of groups responding to consecutive invitations). This, in consequence, could facilitate developing better algorithms of weighting, including improvement of propensity-score weighting methods. Additionally, the study can contribute to developing better data-collection methods in on-site recruitment methodology – used by numerous researchers but with little research and literature on the subject.