Determining how best to collect data in a cost effective manner is crucial. The starting point is to think through the constraints and opportunities for data collection posed by the operational environment while taking into account the resources at your disposal. The purpose of the research is also important. Analysing these factors will help you choose the most appropriate approach to data collection.
In some operational environments with limited infrastructure (e.g. refugee or IDP camps) and weak cell phone penetration, face-to-face surveys are often the only option. Face-to-face surveys generally require a lot of resources for enumerators, supervision and quality control. It is generally easier and less expensive to conduct phone interviews if cell phone penetration is high enough.
The range of tools for face-to-face surveys spans low tech options – like pen and paper – to high tech options such as Kobo Toolbox and other apps, using smartphones and tablets. In politically sensitive environments one has to be careful when considering the use of handheld devices because they may arouse suspicion.
Face-to-face surveys may lead to biases. Courtesy bias is one of the most common, particularly when surveys are conducted by program staff themselves. If NGO staff ask beneficiaries questions about their satisfaction with the services they provide, beneficiaries will naturally be inclined to answer these questions more positively than when asked by an external party. For more detailed information on different types of bias, see the tutorial on Sampling Strategy. These biases can be reduced significantly if addressed adequately as explained there.
Phone surveys are generally less expensive to administer than face-to-face interviews but they only work when the target population has access to cell phones or landlines. For phone surveys targeted at a specific beneficiary group, you will need a list of beneficiaries and their phone numbers. Without this, it is hard to sample randomly. If the target group is large enough, you can make a random sample from the whole subscriber base of a particular telco provider. This works where all members of a community or region are potential beneficiaries. This was the case during the Ebola crisis in Nepal, for example. Phone surveys can be automated or enumerated. Enumerated phone surveys require more resources but allow for more detailed answers, including qualitative information from open-ended questions. Automated phone surveys require a certain level of literacy from respondents and do not allow for collecting detailed responses. That said, automated surveys – especially those administered using SMS – are less expensive because they don't require enumerators.
SMS surveys can be used to survey a random sample of a particular community or region. They can also be targeted at specific beneficiary groups. SMS surveys are best for fast collection of quantitative feedback from a large number of people efficiently. But they require a certain level of literacy, as respondents have to type in their answers. This approach lends itself well to self-collection by agencies. A cheap SMS option, which can be easily managed internally, is Telerivet.
Online surveys are easy to manage but require a high level of technological development, infrastructure and education. Online surveys offer limited space for personal biases (courtesy bias) as they are anonymised. Similar to SMS surveys and automated phone calls, the questions need to be very easy to understand and should not allow for different interpretations. Asking simple and easy to understand questions is key in any survey but is even more important in online surveys. SurveyGizmo or SurveyMonkey are software tools that help manage surveys, collect and analyze data. Online surveys are perhaps best suited for collecting feedback from frontline staff or other similar constituent groups.
A 'survey kiosk' can be a stationary computer or a tablet attached to the wall of a distribution, health or counselling center and allows individuals to give immediate feedback after receiving goods or services or while waiting. The kiosk option predominantly targets beneficiaries and lacks the reach of other survey means. It also requires some beneficiary computer literacy. Apps such as SurveyGizmo and others often have offline kiosk options. The biggest advantage of the kiosk option is that it allows individuals to provide feedback at their convenience as opposed to intentional data gathering exercises.
An experienced, independent data collection group is more likely to reduce response biases and provide high quality data. From the commissioning organization's point of view, meanwhile, collecting data itself may be less costly. Added advantages are that self-collection may be more quickly and sustainably embedded in performance management systems.
Despite these advantages, internal data collection requires training of staff and setting aside resources. In addition to assigning staff as data collectors and training them, supervisors are necessary to ensure quality control. Internal data collection has to be well planned and managed.
Self-collected data is prone to courtesy bias. A study by Keystone Accountability shows that responses were 30% more positive when data was self-collected compared to independently collected data. To reduce courtesy bias, agency staff can take precautionary measures. These include informing responders that candid answers are more likely to influence program delivery and using technology tools that can provide for anonymity (e.g. email survey).
External data collection usually requires less management oversight than internally collected data. That said, briefing enumerators is important in ensuring a successful data collection process. External enumerators need to be aware of the sensitivities and codes of conduct of aid agencies regarding data confidentiality, harassment and so on.
The graph below shows the level of bias and utility for self-collected vs. independently collected data, and for anonymous vs. non-anonymous data.
Can you easily reach your respondents in person?
Yes No
Do you have smartphones or tablets at your disposal?
Is literacy high among your respondents?
Do you need the flexibility to ask optional follow-up questions or open-ended questions to capture specific details?
Do you have access to your respondents' phone numbers?
Do you have access to your respondents' email addresses?
