• How to Collect Data
  • Operational Environment
  • Availabile Resources
  • Face-to-Face
  • Phone & SMS
  • Online & Kiosk
  • Who Collects the Data
  • Bias & Utility
  • Tool Selection

How to Collect Data

This tutorial will guide you through the various considerations for deciding how to collect data and help you select the most appropriate collection option.

Determining how best to collect data in a cost effective manner is crucial. The starting point is to think through the constraints and opportunities for data collection posed by the operational environment while taking into account the resources at your disposal. The purpose of the research is also important. Analysing these factors will help you choose the most appropriate approach to data collection.

Assessing The Operational Environment And The Purpose Of Data Collection

The operational context can enable or limit the use of certain data collection options.

  • Access: Do you have easy in-person access to respondents?
    Face-to-face data collection is often the preferred option, but collecting data this way is complicated in remote locations and where security and violence are issues. These factors may mean remote data collection (i.e. phones surveys) are the only option.
  • Connectivity: How widespread are mobile phones? Do people have internet access?
    The spread of the internet and mobile phones opens up the possibility of using online and phone survey tools, which are described later in the tutorial.  When considering these options, think about whether vulnerable groups, such as women or older people, have access to the internet or mobile phones, and whether choosing these collection options might limit their ability to provide feedback.
  • Literacy: How literate are respondents?
    Literacy is an issue. High levels of literacy mean that respondents can fill out a questionnaire themselves – either online, via SMS or through a kiosk. Low levels of literacy, meanwhile, require enumerators to read the questions outloud, perhaps explain them (if respondents don't understand them), and fill out the survey form.
  • Details: How important is it to have the kind of qualitative information that open questions can provide?
    Depending on the purpose of the data being collected, you might need to ask optional follow-up questions or open-ended questions to capture specific details. If that's the case, options such as SMS surveys or automated phone surveys might not be the best fit for your data collection exercise.

Analysing Available Resources

Taking stock of the resources available can help determine the most appropriate data collection option for your organization.

  • Technical equipment: Do you have smartphones or tablets at your disposal?
    The availability of technical equipment such as smartphones or tablets allows you to use Kobo Toolbox or other similar software to collect and aggregate responses. These tools are inexpensive and relatively easy to use, speeding up data entry and processing.

  • Availability of skilled employees: Do staff have experience in conducting surveys and are they familiar with relevant data collection softwares?
    Conducting surveys either face-to-face or over the phone requires some interviewing skills. If smartphones or tablets are used, data collectors also need experience in using software tools to record responses. Lack of experienced staff means that training is necessary. Some data collection options, such as face-to-face surveys, require a significant amount of staff time. If staff are already stretched, hiring external data collectors may be the best option.

  • Financial resources: What financial resources are available for data collection?
    Financial resources are an important consideration when planning data collection. Available resources will determine whether it is possible to invest in technical equipment, hire additional staff or recruit an independent data collection company to collect the data.

Survey Types: Face-to-face

Face-to-face is the most common type of survey methods in a crisis setting.

In some operational environments with limited infrastructure (e.g. refugee or IDP camps) and weak cell phone penetration, face-to-face surveys are often the only option. Face-to-face surveys generally require a lot of resources for enumerators, supervision and quality control. It is generally easier and less expensive to conduct phone interviews if cell phone penetration is high enough.

The range of tools for face-to-face surveys spans low tech options – like pen and paper – to high tech options such as Kobo Toolbox and other apps, using smartphones and tablets. In politically sensitive environments one has to be careful when considering the use of handheld devices because they may arouse suspicion.

Face-to-face surveys may lead to biases. Courtesy bias is one of the most common, particularly when surveys are conducted by program staff themselves. If NGO staff ask beneficiaries questions about their satisfaction with the services they provide, beneficiaries will naturally be inclined to answer these questions more positively than when asked by an external party. For more detailed information on different types of bias, see the tutorial on Sampling Strategy. These biases can be reduced significantly if addressed adequately as explained there.

Survey Types: Phone & SMS

Phone surveys are good alternatives to labour intensive face-to-face interviews in a setting with high phone penetration and use.

Phone survey (automated/enumerated; random/targeted)

Phone surveys are generally less expensive to administer than face-to-face interviews but they only work when the target population has access to cell phones or landlines. For phone surveys targeted at a specific beneficiary group, you will need a list of beneficiaries and their phone numbers. Without this, it is hard to sample randomly. If the target group is large enough, you can make a random sample from the whole subscriber base of a particular telco provider. This works where all members of a community or region are potential beneficiaries. This was the case during the Ebola crisis in Nepal, for example. Phone surveys can be automated or enumerated. Enumerated phone surveys require more resources but allow for more detailed answers, including qualitative information from open-ended questions. Automated phone surveys require a certain level of literacy from respondents and do not allow for collecting detailed responses. That said, automated surveys – especially those administered using SMS – are less expensive because they don't require enumerators.

SMS survey (random/targeted)

SMS surveys can be used to survey a random sample of a particular community or region. They can also be targeted at specific beneficiary groups. SMS surveys are best for fast collection of quantitative feedback from a large number of people efficiently. But they require a certain level of literacy, as respondents have to type in their answers. This approach lends itself well to self-collection by agencies. A cheap SMS option, which can be easily managed internally, is Telerivet.

Survey Types: Online Survey & Kiosk Option

Online surveys and kiosks can be easy to manage collection options with low variable costs in settings with high internet penetration and computer literacy.

Online survey

Online surveys are easy to manage but require a high level of technological development, infrastructure and education. Online surveys offer limited space for personal biases (courtesy bias) as they are anonymised. Similar to SMS surveys and automated phone calls, the questions need to be very easy to understand and should not allow for different interpretations. Asking simple and easy to understand questions is key in any survey but is even more important in online surveys. SurveyGizmo or SurveyMonkey are software tools that help manage surveys, collect and analyze data. Online surveys are perhaps best suited for collecting feedback from frontline staff or other similar constituent groups.

Kiosk option

A 'survey kiosk' can be a stationary computer or a tablet attached to the wall of a distribution, health or counselling center and allows individuals to give immediate feedback after receiving goods or services or while waiting. The kiosk option predominantly targets beneficiaries and lacks the reach of other survey means. It also requires some beneficiary computer literacy. Apps such as SurveyGizmo and others often have offline kiosk options. The biggest advantage of the kiosk option is that it allows individuals to provide feedback at their convenience as opposed to intentional data gathering exercises.

Who Collects The Data

Data can be self-collected by an aid organization or independently collected by an external party.

An experienced, independent data collection group is more likely to reduce response biases and provide high quality data. From the commissioning organization's point of view, meanwhile, collecting data itself may be less costly. Added advantages are that self-collection may be more quickly and sustainably embedded in performance management systems.

Despite these advantages, internal data collection requires training of staff and setting aside resources. In addition to assigning staff as data collectors and training them, supervisors are necessary to ensure quality control. Internal data collection has to be well planned and managed.

Self-collected data is prone to courtesy bias. A study by Keystone Accountability shows that responses were 30% more positive when data was self-collected compared to independently collected data. To reduce courtesy bias, agency staff can take precautionary measures. These include informing responders that candid answers are more likely to influence program delivery and using technology tools that can provide for anonymity (e.g. email survey).

External data collection usually requires less management oversight than internally collected data. That said, briefing enumerators is important in ensuring a successful data collection process. External enumerators need to be aware of the sensitivities and codes of conduct of aid agencies regarding data confidentiality, harassment and so on.

Determining the Levels of Bias and Utility

The graph below shows the level of bias and utility for self-collected vs. independently collected data, and for anonymous vs. non-anonymous data.

Find the tool that is right for you

What best describes the resources and capacities of your organization?

Can you easily reach your respondents in person?

Do you have smartphones or tablets at your disposal?

Is literacy high among your respondents?

Do you need the flexibility to ask optional follow-up questions or open-ended questions to capture specific details?

Do you have access to your respondents' phone numbers?

Do you have access to your respondents' email addresses?