Survey Project Design

Learn how to establish survey objectives, methods, institutional review and approval, and sampling to create a successful project.

Challenge

Designing a survey project can be a daunting task. How do you establish survey objectives, methods, institutional review and approval, and sampling to create a successful project?

Solutions

Establishing Survey Objectives

Before you begin collecting data, you should determine what type of survey you are conducting, adequately research your topic, and define your objectives.

Determine what type of survey you are conducting.

Are you conducting research or evaluation? The type of survey you choose to conduct will impact your objectives, methods, sampling, and analysis. Below are the descriptions of research and evaluation.

Research: is the process of gathering, analyzing, and interpreting data to describe the current level of institutional or departmental effectiveness. Research objectives and sample sizes are broad in nature. The results gained from research often add information to an existing field or contribute information to theory development tests. These inferences can then be applied to a variety of situations. For example:

  • A survey of students’ experiences and attitudes toward healthy eating and physical fitness.
  • A survey about a population’s experiences, attitudes, and perceptions regarding sexual harassment.

Evaluation: uses evidence found through research to improve institutional or departmental effectiveness. The objectives and samples used in evaluation projects are clearly defined compared to research objectives. Evaluations provide a snapshot of what is currently happening in a specific program and how that program can be improved. Common evaluations are:

  • Customer satisfaction survey
  • Conference or event evaluation

Gather Information

  • Review existing literature on your topic to see how it has been framed by other researchers, noting what information was included or left out.
  • Consider other related issues or themes when exploring a specific topic as they might prompt additional questions.

Define the objectives of your study.

  • Define the research objectives you want to address.
  • Meet with colleagues and stakeholders to elicit their thoughts on important issues to address.
  • Consider whether you are conducting research or an evaluation to determine whether your objectives will be attainable for your survey type. Research objectives are broad in nature and its results offer new information to a field of study, while evaluative objectives are clearly defined and its results give specific answers.
  • Transform your objectives into the questions you want to be answered.

After you have addressed the preliminary aspects of project planning, you are ready to move on in the survey planning process and decide what method of survey is right for you.

Methods

There are three methods used to conduct survey research: quantitative methods, qualitative methods, and mixed methods.

Below is a look at the different methods of data collection and a list of their strengths to help you decide which meet your research needs.

MethodDescriptionStrengths

Quantitative

  • Paper surveys

  • Web surveys

Focuses on deduction, confirmation, theory/hypothesis testing, prediction, standardized data collection, and statistical analysis.
  • It provides precise, numerical data that is easy to analyze.

  • Useful for studying large numbers of people.

  • Data analysis is relatively less time consuming (using statistical software).

  • Can generalize research findings when the data is based on random samples of sufficient size.

  • The results are relatively independent of the researcher.

Qualitative

  • Focus groups

  • Interviews

Focuses on induction, discovery, exploration, theory/hypothesis generation, and qualitative analysis. The researcher is the primary ‘instrument’ of data collection.
  • It provides an understanding and description of people’s personal experiences.

  • Useful for describing complex phenomena.

  • Interviewers can be successful in eliciting further information from the respondents through careful probing.

  • Useful for studying a limited number of cases in-depth.

MixedA mixture or combination of quantitative and qualitative methods in a single study.
  • Qualitative and quantitative research used together produces more complete knowledge.

  • It can add insights and understanding that might be missed when only a single method is used.

  • It can provide stronger evidence for a conclusion through convergence and corroboration of findings.

  • Numbers can be used to add precision to words, pictures, and narrative.

  • Words, pictures, and narrative can be used to add meaning to numbers.

Institutional Review and Approval

The University of Minnesota is dedicated to conducting research with integrity. As a result, all research projects must adhere to strict ethical standards of the University, in addition to regulations and procedures administered by local, state, and Federal governments. The Institutional Research Board and the Human Research Protection Program work together to review all research projects to ensure they meet every standard.

  • Institutional Research Board (IRB): assesses the risks and benefits of a research study before it begins in order to protect the project’s subjects. If the benefit outweighs the risk involved, the IRB initializes a consent process to ensure subjects are fully aware of the project’s risks and benefits.
  • Human Research Protection Program (HRPP): processes IRB applications to review proposed research projects and related issues. They also assist independent research review committees and executive committees from other panels.

Below is an overview of the full-committee review process all research projects must undergo.

  1. You must submit an application form to the IRB along with your consent forms and other required appendices/materials.

  2. The staff at the HRPP will pre-review your application to make sure all of the necessary materials are available for the IRB review.

  3. Once all of the materials necessary for an IRB review have been collected, your study will be assigned an IRB number, a committee number that will review your study, and the date your study will be reviewed.

  4. Two weeks after the application deadline the IRB will convene and make a decision about your application.

  5. Seven to ten days after the meeting you will receive a letter from the HRPP detailing the outcome of the IRB meeting in regards to your research.

It is important to note that not all research projects need clearance from the Institutional Research Board. Examples of projects that do NOT need IRB review include:

  • Searches of existing literature.
  • Quality assurance activities or evaluation projects designed for self-improvement or program evaluation. These projects are not meant to contribute to “generalizable” knowledge.
  • Interviews of individuals where questions focus on things, not people. For example, questions about policies.

Examples of projects that DO require IRB review include:

  • Research involving human subjects
  • Using records gathered on human subjects
  • Involving human tissue

Contact HRPP for more information if you:

  • Are uncertain whether or not to apply to IRB
  • Need more information about the full-committee and expedited review processes
  • Want clearance from the application process


 

Sampling Best Practices

A sample is used to randomly identify a subgroup of faculty, staff, students, alumni, and other stakeholders that the researcher plans to study for generalizing different populations at the University of Minnesota.

It is inappropriate to send out multiple survey invitations and reminders to a large number of recipients. This leads to low response rates and institutional survey fatigue, which negatively impacts data quality across the University. This section will provide information for you to consider when pulling or requesting a sample.

Representative Sampling

A representative sample is when a sample is an accurate, proportional depiction of the population under study. There are two techniques used to attain representative samples: randomization and stratification. Below is an example of each.

Randomization

If you want to study the attitudes of U of M students regarding student services, it would not be enough to interview every 100th person who walked into Coffman Memorial Union. That technique would only measure the attitudes of U of M students who go to the union, not those who do not. In addition, it would only measure the attitudes of U of M students who happened to go to the union during the time you were collecting data. Therefore, the sample would not be very representative of U of M students in general. In order to be a truly representative sample, every student at the U of M would have to have had an equal chance of being chosen to participate in the survey. This is randomization.

Stratification

If you took a list of U of M students, uploaded it onto a computer, and then instructed the computer to randomly generate a list of students, your sample still might not be representative. What if, purely by chance, the computer did not include the correct proportion of seniors or graduate students? If these fields are of interest to you and you want the sample to be more representative, you might want to use a sampling technique called stratification.

In order to stratify a population, you need to decide what sub-categories of the population might be statistically significant. For instance, graduate students as a group probably have different opinions than undergraduates regarding student services, so they should be recognized as separate strata of the population. Once you have a list of the different strata, along with their respective percentages, the sample would be ensuring that a certain percentage is graduate students and a certain percentage is seniors. You would then come up with a more truly representative sample.

Controlling Marginal Error

Anytime you survey a portion of a population there will be some margin of error in the results. However, you can control your level of error mathematically by using a specific confidence interval and sample size.

To do this, you must:

  1. Define the size of the target population.
  2. Determine your desired level of error.
  3. Determine your desired level of confidence.
  4. Calculate the sample size.

The level of error is measured as a percentage, as is the level of confidence. The level of confidence represents how confident you feel about your error level. For example, if you have a 95% confidence interval with an error level of 4%, you are saying that if you were to conduct the same survey 100 times, the results would be within +/- 4% of the first time you ran the survey 95 times out of 100.

The tables below give examples of different sample sizes at a 95% confidence interval.

Survey level of error interval example

 

Custom Insight offers an online Survey Random Sample Calculator that calculates how many respondents are needed for a survey, how many people to send a survey to and the accuracy of survey results.

Duplication

The most common sampling error is duplication or instances where one of the target population elements is overrepresented. This can produce biased survey results.

There are three ways to reduce or eliminate duplication as a source of sampling error.

  1. Eliminate duplicates from the sample. This can be done by sorting the samples and removing any duplicate information through Microsoft Excel.
  2. Establish a rule for handling duplicates. This occurs during selection and/or data collection. One example rule could be using only the first entry on the list, then identifying and removing its duplicate.
  3. Weight responses are known to be duplicated. When analyzing a sample, a researcher can weigh the duplicates to ensure that these elements are not overrepresented in the data without eliminating responses. For example, if there are four known duplicates, each could be weighted by .25.