Survey-based research: General principles

  • Author Footnotes
    1 Edward A. Panacek, MD, MPH, is professor of emergency medicine and clinical toxicology at the UC Davis Medical Center in Sacramento, California.
    Edward A. Panacek
    1 Edward A. Panacek, MD, MPH, is professor of emergency medicine and clinical toxicology at the UC Davis Medical Center in Sacramento, California.
    Search for articles by this author
  • Author Footnotes
    1 Edward A. Panacek, MD, MPH, is professor of emergency medicine and clinical toxicology at the UC Davis Medical Center in Sacramento, California.
      Surveys are a well-established and commonly employed category of research study design. A review of prior issues of Air Medical Journal and many other clinical journals usually reveals multiple articles that used a survey design format. Surveys, as a form of research, seem to be particularly popular among novice researchers. Unfortunately, many of those novice surveys are not well designed, and, as a result, surveys research has garnered a questionable reputation. Indeed, some journals (eg, American Journal of Emergency Medicine) will not even consider survey-based research for publication.
      However, when done well, surveys are a perfectly legitimate form of research. Indeed, some research questions could be addressed only by using a survey design. In addition, there is well-established and validated science to support the legitimate use of surveys in gathering investigative information. That science can be rather complex, and some researchers spend their entire professional career focusing entirely on surveys. However, the basic principles are relatively straightforward and can be applied even by novices. This article covers some of those basic principles.
      The term survey includes any studies that consist of asking individuals to respond to questions. This includes written questionnaires and personal interviews. These can be formal, informal, anonymous, internet-based, etc. There are certainly important differences between these different techniques, but this article does not have sufficient space to address each category individually. For the purposes of this discussion, survey will generally be used to refer to all of these different designs.
      As a research tool, surveys have a number of important advantages. First, they are relatively inexpensive to perform. Second, they often allow quick data acquisition. Third, an appropriately drawn sample from a target population can provide accurate and representative data. Accurate results are certainly the goal of any research project, including survey-based research. There are some basic principles that apply to surveys that increase the likelihood of obtaining accurate and meaningful data. They are listed in Table 1 as the “Ten Commandments of Performing Survey-Based Research.” Each one of those principles will be discussed in order. Poor quality survey research, performed by novice investigators, generally fails to apply these 10 principles. Because of that, they often end up with results that are not valid.

      Be clear on the purpose of the study

      All research projects start with the all-important primary research question. Surveys are no different from any other form of research in this regard. The principles of developing a properly formatted and well-articulated research question have already been discussed in an earlier part of this series. Those principles should be applied to even the simplest survey research project. Once the research question is well understood and has been refined, the question is whether a survey study design is the most appropriate way of obtaining the data to best answer that question. For some questions, the answer is clearly yes, and it is appropriate to proceed with a survey study design. However, for many other questions, a survey is simply an expedient way of trying to obtain data, not necessarily the only or even most scientifically valid approach.

      Focus the entire survey on the research question

      Research projects can occasionally have some secondary research questions, but the primary research question must always be kept paramount. It is very easy for surveys to expand in size, length, and scope. However, the probability of survey completion and good response rates is inversely proportional to survey length; therefore, it should stay as small and focused as possible. What is the minimum number of questions or data points needed to adequately answer the primary research question? That should be the focus of the survey. Drop peripheral or redundant questions, whenever possible.

      Use prior survey techniques, whenever possible

      Developing a new survey instrument from scratch is a much bigger task than most novice researchers realize. Decisions regarding the format, structure number, and order of survey questions can be quite complex. Most beginners do not understand the principles that underlie valid survey question design. Access to an experienced survey researcher can be very helpful in this regard. However, most novices do not have access to such individuals. Instead, you can attempt to learn and borrow from the work of others in similar or related fields.
      Ten Commandments of Surveys Research
      • 1.
        Start with a clear-cut research question
      • 2.
        Keep the survey focused and user friendly
      • 3.
        Use prior research methodology and questionnaires whenever possible
      • 4.
        Decide on open versus closed question formats
      • 5.
        Validate the survey instrument
      • 6.
        Pilot test the survey instrument
      • 7.
        Use proper sampling techniques
      • 8.
        Plan for low response rates
      • 9.
        Adjust for nonresponders
      • 10.
        Do not overinterpret study results
      This inexperience underscores the importance of performing a quality literature review in the field. Articles often can be identified that include copies of the questions or entire survey instruments that were used in prior studies. Sometimes the survey instrument used in a given study is posted on the journal website. Those questions can often be borrowed, modified, or even directly applied to a new survey project. In addition, the descriptions of research methodology used in prior survey studies can be quite helpful in designing your own project.
      Be aware that some questionnaires or survey instruments are proprietary and copyrighted. However, most of those published in the medical literature are available for modification and use. Giving appropriate citation reference and credit to the original authors avoids issues of plagiarism. In addition, minor or even moderate modifications are often required to apply previously used survey questionnaires to a new project. For example, a patient satisfaction survey developed and used in an outpatient clinic would likely not be directly applicable to surveying air-transported patients without making some modifications to the questions. Nonetheless, the general tone, number, and organization of the questions may still be relevant to the air medical researcher.

      Decisions regarding question structure

      In surveys, there are two main categories of questions—open-ended and closed. Closed questions give the subject only a limited selection of potential answers. They force them to choose one option. Open-ended questions are free form and allow the subject to provide any answer he or she wishes. There are advantages and disadvantages to each of these formats. In general, closed questions are preferable because they simplify the process of analyzing the results. Most commonly, each of the answer response rates are summarized with simple descriptive statistics. Such questions provide numeric data that can be summarized, reported, and sometimes compared. In general, closed questions are considered more scientifically valid in terms of the data generated. However, the investigator must understand all of the relevant or appropriate responses for a given question, so that each of those options is included. If important and relevant answers or answer categories are not included in the options, the results could be quite biased.
      Open-ended questions do not have this problem. They allow for a full range of responses. The difficulty is in analyzing the answers. Sometimes free text answers can be mapped into prescribed categories and then summarized and analyzed. Other times, that becomes more difficult. In general, when an investigative field is relatively immature and the full range of relevant answers are not well understood, at least some open-ended questions are recommended. However, as a field becomes more mature, closed questions are recommended. Although open-ended questions can be hard to analyze quantitatively, they often provide very useful quotations. These can be quite powerful, particularly in the fields of politics and advocacy. Having free-form comments to quote can make for useful sound bites on the news, in speeches, or in lectures.

      Validate the survey instrument

      The term validity is the scientific design term for accuracy. In other words, does the measurement tool accurately measure what is intended? All studies measure something. Any survey instrument is a measurement tool designed to collect data accurately and reliably. However, unlike mechanical instruments such as blood pressure monitors or a laboratory chemistry machine, the validity of a survey instrument is not automatically assumed. It is the investigator's responsibility to establish the validity of the survey instrument being used in their respective study.
      There are a number of different approaches to establishing validity. However, the easiest is to use or adapt an already validated survey. As mentioned, performing a quality literature search on the subject is important for multiple reasons. It provides guidance and insight into performing survey-based research and study methodology. Just as important, it may provide an actual survey instrument that can be directly used or modified for the purposes of the new investigation. Once a survey questionnaire has already been used in a prior published study, it is considered to be at least generally validated. If indeed it appeared to measure its intended target, it is accepted to be valid from a peer review standpoint. Therefore, use previously published survey instruments whenever possible. If they require some modification, attempt to keep those changes at a minimum and be prepared to argue that those changes did not substantially alter the survey instrument's overall validity.
      Often, there are no previously published survey instruments or questionnaires that are directly applicable to the investigator's intended project. It then is the responsibility of the investigator to validate the new survey instrument. Five general categories of validation are considered in such situations (Table 2). The simplest of these approaches is called face validity. In other words, by simply glancing at the “face” of the survey instrument, does it appear to make sense and be relevant to the research project? The individual assessing the survey's face validity is generally the investigator co-investigators. However, such individuals may be considered biased. As a result, face validity is the most questionable approach to establishing validity. Many research methodologists consider it to be an inadequate approach. As such, it is not generally recommended. Instead, the investigator should attempt to use another category.
      Categories of Validation for Survey Instruments
      • Face validity
        • Does it make sense simply by looking at it?
      • Content validity
        • A panel of experts reviews and revises it for relevant content.
      • Criterion validity
        • How does it compare with an established standard in the field?
      • Predictive validity
        • How does it compare to actual outcomes of interest?
      • Construct validity
        • Does the internal construct/structure conform to scientific survey study principles?
      Content validity is similar to face validity but taken to the next level of sophistication. It employs a panel of experts in the field who review the survey instrument for content and structure to decide whether it addresses the research project's goals. Because it uses multiple individuals and purported experts in the field, it is considered a higher level of validity than simple face validity. Many new survey instruments are validated through this approach. Pilot testing is also commonly used to help establish content validity. If pilot testing finds content problems, the questions are adjusted as needed.
      Criterion validity is considered a yet higher level of validity than face or content validity. It employs an already well accepted “gold standard” measurement instrument in the field and compares the newly developed survey instrument with that standard. For example, imagine an investigator who wishes to measure the prevalence and severity of anxiety amongst helicopter-transported patients. That investigator finds an already well-established psychiatric survey questionnaire that is known to accurately measure those variables. However, it is a 10-page questionnaire and not very practical for the air medical environment. The investigator develops a new, simple, 1-page, 5-question instrument. Both are administered to air-transported patients, at the same time, and the results are compared. The question is whether the shorter new instrument compares favorably to the well-established criterion standard. If it gives essentially the same results, it is considered criterion validated.
      Predictive validity is most applicable in instances in which a survey or similar instrument is being used to predict outcomes or severity. For example, a simple survey questionnaire or simple 1-page scoring system could be used in patients presenting with chest pain that may be an acute myocardial infarction. It is designed to predict the severity of illness and mortality rates. It would be applied prospectively to all such patients. Those patients would be followed for their actual outcome in terms of morbidity and mortality. The survey or other measurement instrument would be directly compared with actual patient outcomes to see whether it accurately predicts them. This is a highly scientific approach to validating such a measurement instrument.
      The last category and one more specifically directed toward survey instruments is that of construct validity. This is a complicated subject best left to researchers very experienced in survey design. It should not be used by novice researchers in the field. In brief, it focuses on the overall structure of the survey and often uses multiple related questions within the same survey questionnaire. It then looks for consistency across answer patterns by using complex analytic techniques.

      Properly sample the population

      All studies are sampling a subset of the overall population. No investigation can ever study the entire universe of potential subjects, so it is always selecting a subsample from that universe. Surveys do the same thing. There are differences depending on whether a survey is performed by mail, by telephone, or in person, but general principles always apply. The goal is always to obtain an adequate and representative sample of potential subjects. That way, the final results are meant to be truly reflective of the target population to which they would apply.
      A number of different approaches to sampling were well discussed in Part 5 of this series, so they will not be repeated here. Suffice it to say that any survey research project intends to obtain an adequate and representative sample of study subjects. Validity of the results depends highly on achieving that goal.


      A large body of literature supports and describes highly scientific methods of performing surveys research. This brief article is simply an introduction to some of the basic principles. The next part in this series will cover more specific issues related to performing a survey and analyzing the results. It will also provide suggested readings. For a more in-depth discussion of this topic, the reader is referred to those references.
      Most importantly, the researcher needs to understand that, although surveys are considered a relatively easy and simple form of research, there are highly scientific principles that underlie the proper performance of such research. Together, these two articles attempt to emphasize those principles while also identifying common pitfalls.