Advertisement

Reviewing the literature

  • Author Footnotes
    1 Cheryl Bagley Thompson, RN, PhD, is an associate professor, Director of the Health Informatics Program, and Assistant Dean for Informatics and Learning Technologies at the University of Nebraska Medical Center, College of Nursing, in Omaha, Nebraska
    Cheryl Bagley Thompson
    Footnotes
    1 Cheryl Bagley Thompson, RN, PhD, is an associate professor, Director of the Health Informatics Program, and Assistant Dean for Informatics and Learning Technologies at the University of Nebraska Medical Center, College of Nursing, in Omaha, Nebraska
    Search for articles by this author
  • Author Footnotes
    2 Edward A. Panacek, MD, MPH, is professor of emergency medicine and director of clinical trials at the UC Davis Medical Center in Sacramento, California
    Edward A. Panacek
    Footnotes
    2 Edward A. Panacek, MD, MPH, is professor of emergency medicine and director of clinical trials at the UC Davis Medical Center in Sacramento, California
    Search for articles by this author
  • Author Footnotes
    1 Cheryl Bagley Thompson, RN, PhD, is an associate professor, Director of the Health Informatics Program, and Assistant Dean for Informatics and Learning Technologies at the University of Nebraska Medical Center, College of Nursing, in Omaha, Nebraska
    2 Edward A. Panacek, MD, MPH, is professor of emergency medicine and director of clinical trials at the UC Davis Medical Center in Sacramento, California
      In the first part
      • Thompson CB
      • Panacek EA
      Basics of research part 1: clinical research and critical care transport: how to get started.
      of the Basics of Research series, you learned how to get started in research and formulate an appropriate research question. The next step in developing a research project is to perform a comprehensive review of the relevant literature, defined by Burns and Grove as “an analysis and synthesis of research sources to generate a picture of what is known and not known about a particular situation or research problem.”
      • Burns N
      • Grove SK
      The fundamental premise of the research question and subsequent selection of an appropriate study design depends on what gaps exist in current knowledge. These deficiencies must be assessed accurately to prevent wasting time, energy, and resources in addressing a question that has been answered already. Thus, you become an expert on the current state of knowledge for the subject area.
      The literature search is an organized method of identifying published material to review that contains scientific information concerning your research question. The primary focus of a literature search is on journals, but additional information may be found in textbooks, conference proceedings, monographs, websites, systematic reviews, and dissertations, among others. Journals are regarded as the principal source of knowledge in the medical community
      • Miller RA
      • Groth T
      • Hasman A
      • Safran C
      • Shortliffe EH
      • Haux R
      • et al.
      On exemplary scientific conduct regarding submission of manuscripts to biomedical informatics journals.
      and have significant advantages over other information sources. Their frequency of publication and shorter preparation time makes them a more up-to-date resource than textbooks or monographs. The strength of the peer review process may increase the quality of the manuscripts
      • Schroter S
      • Tite L
      • Hutchingss A
      • Black N
      Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors.
      when compared with other research-related sources. The availability of online publications makes journal articles more readily available than conference proceedings or dissertation abstracts.
      Research manuscripts in peer-reviewed journals are often considered the gold standard. Peer review means that, after the paper is submitted, individuals with either knowledge or experience concerning the topic of interest critique the study. This serves two basic functions. First, poorly designed, executed, or written material is rejected and kept out of print. In actuality this works better as a concept than in practice,
      • Smith R
      The trouble with medical journals.
      but it does mean that the paper must have cleared at least one hurdle before publication. Second, the reviewer may make suggestions to the authors to help improve the manuscript, such as clarifying certain sections, reorganizing, adding or deleting data, or making other changes designed to improve the project. Thus, peer-reviewed research publications should be your primary source for background information related to your research topic.

      Literature resources

      In 1995 Davis, Thompson, and Panacek
      • Davis EA
      • Thompson CB
      • Panacek EA
      Basics of research part 2: reviewing the literature.
      noted, “Searching the literature is much easier today than it was 5 to 10 years ago,” and the same can be said today. Even further strides have been made in the past 11 years in making the biomedical research literature available to clinicians and researchers alike. Long past are the days of slow searches through large books such as the Index Medicus or Cumulative Index of Nursing and Allied Health Literature (CINAHL) which indexed health care-related literature by subject heading. Also relegated to history is accessing research via tools such as CD-ROMs maintained at a local library, online subscription services, or pay-for-use access to the National Library of Medicine. Rather than being limited to a few pay-for-use services, researchers are able to search the entire Web, often for just the cost of an Internet service provider. Even those charges can be avoided by searching from places such as a medical or public library.
      Unlike the mid-1990s, many tools for searching the biomedical literature are now free once the user is connected to the Internet. Multiple search engines such as Google, Yahoo, and Dogpile are freely available and can be used to search for biomedical research. The primary limitation of these tools is the retrieval of much more data than you could possibly use. For example, a recent search of Google for the words “air transport” and “myocardial infarction” yielded 12,600 references. Although many of these may be of high quality, finding the key references may be challenging, given the volume of options.
      In contrast, PubMed (www.pubmed.gov), a resource similar to the older Index Medicus, limits searches to primarily the formal biomedical literature. PubMed is one resource of the National Center for Biotechnology Information (NCBI) of the National Institutes of Health and contains references back to the 1960s. In addition to public databases, NCBI provides software tools for analyzing genome data and distributes other biomedical information (www.ncbi.nlm.nih.gov). Because PubMed searches fewer, targeted sources, the results will usually be more limited and of greater relevance to your topic. For example, the search described above yielded only nine references on “air transport” and “myocardial infarction.”
      Medical libraries may offer additional search engines specific to the biomedical and related literature. EBSCO Information Services, CINAHL, and PsychInfo are only a few of the databases that may be available. EBSCO has a broad focus, not limited to healthcare, and may uncover resources not otherwise indexed. CINAHL's primary focus is on the nursing and allied health literature. This database currently goes back as far as the 1930s. Many but not all of CINAHL's references can be found in PubMed as well. However, CINAHL also includes lesser-known nursing and allied health journals and some non-journal literature such as nursing texts, conference proceedings, and dissertations. Thus, a search of both databases may be of value.
      • Allen P
      • Jacobs SK
      • Levy JR
      Mapping the literature of nursing: 1996-2000.
      PsycINFO contains references to the psychological literature dating back as far as the 1800s (www.apa.org/psycinfo/about). Note that both PubMed and CINAHL index Air Medical Journal.

      Searching techniques

      Once you have identified an appropriate database or databases, the next step in your search strategy is to identify key words to use in the search. If you have available a list of MeSH, (medical subject headings) terms, it is best to search through this list to find those most closely related to your research question. MeSH terms are a constrained vocabulary developed by the National Library of Medicine to index articles from biomedical journals and other related documents contained in the library.
      • National Library of Medicine
      Fact Sheet: medical subject headings (MeSH).
      Most biomedical databases are indexed by MeSH terms, so your search will be more fruitful if you can use one of the included terms rather than another similar term. You may search the complete list of MeSH at www.nlm.nih.gov/mesh. Indexes also can be searched by non-MeSH terms. These searches are less powerful and do not take advantage of features such as synonym searching.
      More than one search term is usually needed to obtain a usable list of references. Most single-term searches are broad and retrieve more articles than will be useful. A better approach is to use two or three related terms that narrow your topic. For example, if you are interested in management of neonatal pneumothorax during transport, you could use the terms neonatal and pneumothorax. Without the use of both terms, you could get all articles on pneumothorax or all on neonates, neither of which would be helpful because of the large number of citations and lack of specificity.
      Note that the term transport has been deleted from the above list of search terms. Just as too few terms may yield too many articles, too many terms can yield too few articles. If you retrieve too many articles with the first two terms, you can consider adding a third term (eg, transport). In all likelihood, an article on neonatal pneumothorax need not be limited to care of patients during transport to be of value.
      In the past, few individuals had adequate knowledge of proper search techniques so they often sought the assistance of a librarian. Medical librarians are familiar with available resources, health care terminology, and the process for searching through this subset of literature. Medical librarians can assist you in doing your own search or conduct the search independently for you. Although many individuals are now comfortable with the process of searching electronic databases and conduct their literature review independently, a medical librarian may still be of assistance if insufficient time is available for a search or if the researcher does not have direct access to many of the needed references.
      Experience has demonstrated that not all references of relevance will be found with a literature search. The reasons for this are not always clear, but an investigator must be alert for other articles of relevance. These “hidden” references most often appear as citations in the manuscripts the researcher has identified. A quick review of the reference list for all retrieved articles is recommended as a double check for locating relevant material.

      Literature retrieval

      Once you have conducted the literature search and have a list of potential references, a more formidable task awaits—how to decide which articles to retrieve. The search, in most cases, generates a large number of articles, and you must then reject unsuitable sources so that you can concentrate on those few that may truly impact your project.
      Most reference databases will provide a copy of the abstract along with the full citation (title, author, year, journal, volume). Much of the information necessary to identify relevant articles can be determined from either the title or the abstract. First, determine whether the article is a review article or a presentation of original research. Both types of articles may be of value. Review articles provide an overview of the topic. These articles are good for obtaining a summary of the work done to date in the field. These articles may help to confirm that your research question still has not been answered, while describing what is known about the question.
      However, review articles are considered secondary sources. They represent the author's or authors' interpretation of someone else's research. In contrast, a primary source is an article discussing a specific study, written by the author or authors who conducted the study. In general, a primary source provides more specifics as to how the study was conducted and is not subject to potential errors of interpretation by a second party. Primary sources are preferred whenever possible.
      • Burns N
      • Grove SK
      An additional review criterion that may be helpful in determining whether to retrieve an article is the list of authors. One or more of the authors may be familiar to the searcher, and their track record may be known. If their record is positive and stood the test of time, the article may be of increased value. The converse is also true. Less weight should be given to authors with a less-than-stellar reputation. It may become apparent that an author or group of authors has published a large body of work on a particular subject and may be considered experts in the field. Often, however, most authors will be unfamiliar to you. To a certain extent, consider the reputation of the journal as well; but this is not foolproof, either.
      Once you have identified a list of articles that you have determined are relevant to your project, you must retrieve the articles so that you can read them in their entirety. Although searching through the literature for items of interest has become infinitely easier, retrieving articles still remains somewhat problematic. Electronic copies often are preferred, as the researcher need not actually go to the library to obtain a copy.
      Many journal articles and other electronic sources are now freely available though the Web. Some journals are available immediately on publication; others remain restricted for a period, often ranging from 6 to 12 months. One advantage of searching PubMed is that many citations are accompanied by a link to an electronic source for the article. If the article is not freely available, the user usually will be provided with a method whereby a copy of the article can be purchased. The price for articles can be expensive, however, as high as $25 per article in some cases. This fee may provide the user with a PDF copy of the article or access to the article (or even the entire issue) online for a given period.
      If an electronic copy of the desired article is not available, a hard copy may be found at a local library, particularly a medical library. If the library subscribes to the journal in print format, you can usually make a copy to take home by using copy machines found in the library. In other cases the library may subscribe to the journal electronically, providing users the ability to read the article online while in the library or even print a copy to take with them. If the library does not have either an electronic or hard copy of the journal, they will in most cases be willing to order one for you via interlibrary loan. Charges for interlibrary loans vary. If you are employed by a hospital, the hospital library may provide the service free.

      Literature review

      Although an article may appear to be of value from the title or abstract, the entire article must be read to determine relevance to the proposed study. Several points of review should be applied to each article to determine its suitability. You may find it helpful to first read the summary or conclusion. The purpose is to determine whether the results of the paper, if valid, are relevant to your research question. At this stage, determining the validity of the results is not your primary purpose. Rather, the goal is to decide which results, if true, would be useful in designing your study.
      Next, determine whether the patient population and the circumstances under which the study took place are relevant to your intended project. Ideally the subject in the article will be similar to your intended study population with respect to such variables as age, gender, disease, prior treatment, and so forth. Studies in related populations also may be of value. For example, if findings of the effectiveness of treatment A were found in female trauma patients but not studied in men, this would provide justification for your study in men. A final point to note in the paper's conclusions/summary is the limitations identified by the author. The limitations of one study often provide suggestions as to how to improve the next study—in this case, yours.
      If an article has passed the above basic scrutiny, the full article should undergo a detailed review. A key purpose of the review, once applicability to your research question has been confirmed, is to determine the validity of the findings. This process varies with the design or intent of the published study. Most medical articles can be placed into one of the following categories, each with a differing set of criteria for scientific value:
      • 1.
        Evaluation of a new therapy
      • 2.
        Evaluation of a new diagnostic test
      • 3.
        Determination of the cause of a condition
      • 4.
        Prediction of the outcome or natural course of a condition
      Articles dealing with evaluation of a new therapy should be approached in the following manner. With respect to results, the magnitude of the treatment effect should be explored. The larger the difference between individuals receiving therapy and individuals who have not, the more clinically significant the treatment is likely to be. The accuracy of the measurement of effect is also important; the more precise and reliable the measurement, the greater the confidence in the results. Accuracy is influenced by the quality of your measurement results, instrument, and the appropriateness of your measure. The number of trial participants affects precision. Larger studies provide a more focused estimate of effect.
      Validity, defined as whether the measured effect represents the true direction and magnitude of the treatment effect,
      • Burns N
      • Grove SK
      is affected by a number of conditions. The participants should be randomized properly to treatment and control groups, and the process must not be influenced by outside factors. Treatment and control groups should be equal in all aspects except the experimental therapy to prevent any confounding effect on outcome. Ideally, both the subjects and those administering treatment are blinded to what therapy is used on each individual (double-blind study). This prevents any conscious or unconscious manipulation of the results by the subjects or investigators.
      All patients entered into the study should be accounted for at the end. If a large number of subjects are lost, the reported findings may not be valid. Subjects who complete a study may differ in meaningful ways from those who do not complete a study. Thus attrition may influence the findings, especially if not evenly spread between control and experimental groups.
      • Woolard RH
      • Carty K
      • Wirtz P
      • Longabaugh R
      • Nirenberg TD
      • Minugh PA
      • et al.
      Research fundamentals: follow-up of subjects in clinical trials: addressing subject attrition.
      Attrition can be particularly problematic in emergency medicine, because the clients have no ongoing connection to the emergency department for continuing care.
      A second type of study is evaluating a new diagnostic test. Studies involving the evaluation of a new diagnostic or other test may be relevant if you are investigating the same test or if you plan to use this new approach for measurement in your study. Studies concerned with diagnostic tests should be evaluated for likelihood ratios, or how well the test predicts the presence or absence of a condition in an individual patient.
      • Honest H
      • Khan KS
      Reporting of measures of accuracy in systematic reviews of diagnostic literature.
      Obviously, the higher the probability that a positive test is associated with the disease state and that a negative test occurs in the patient without the condition, the more valuable the test. The test should be evaluated by a blind and independent comparison to an accepted gold standard, ideally the diagnostic test in most common use for the suspected disorder. The trial should apply the test to a wide variety of patients, covering as much of the clinical spectrum in which the investigator is interested as possible.
      The potential effect on the performance of the current gold standard should be addressed. If the new test adds little to the current method of evaluation, is not less expensive, or is not more time efficient, it is unlikely to have a substantial clinical impact. The new test must be able to be replicated easily by those planning on using it in either a clinical or research context.
      Prediction of the cause of a condition is a third type of study concerned with the issue of cause and effect. The results should be viewed with respect to the strength of exposure that results in a particular outcome (ie, how often the exposure results in the patient being affected by the outcome of interest). As previously discussed, the precision of the risk estimate should be evaluated. Validity is increased if the groups being studied were as similar as possible in all respects affecting outcome except the risk factor being studied. Commonly this is reported in the results section as the “baseline” parameters or demographics and deals with such factors as age, sex, race, etc.
      Proper randomization procedures are preferred to limit possible sources of bias, and the groups being compared must have their exposures and outcomes measured by the same method. Randomization of some exposures (eg, motor-vehicle accident, cocaine use) cannot always be achieved. In these cases, equivalence of other baseline parameters takes on increased importance.
      When addressing issues of cause, authors need to demonstrate that the proper temporal sequence exists. The exposure must precede the outcome, and the strength of the association is enhanced by a dose-response relationship. This means that as the magnitude or duration of the exposure increases, so does the occurrence of the outcome in question. The patients must be followed for an appropriate period to ensure capture of all relevant events. An exposure that results in an outcome decades after the fact will obviously not be addressed by a 5-year follow-up period.
      Once again, compare the study and your clinical population to determine whether the results can be extrapolated to your situation. The clinical importance, or magnitude of the risk, should be addressed to determine whether manipulation of the environment is warranted to prevent the undesirable outcome.
      The fourth grouping of articles, those concerned with predicting outcome, use many similar review points as the previous sections. The results are evaluated by the magnitude of outcome likelihood or how large the occurrence of events is within a specified period. Precision must be addressed. Validity is judged by determining whether a patient sample is truly representative of the entire population in question, and whether sufficient follow-up occurred. The criteria for judging the outcome criteria must be unbiased and objective to prevent the researchers from influencing the true result. Similarity of the study patients to the population in which you are interested should be considered. Most important is the determination of whether the results of the study lead to changes in therapy or other practices.
      Several general points apply to every review process. Good papers will discuss their own limitations, usually in one of the last few paragraphs before the conclusion. Pay attention to this discussion, as small problems may have a large effect on the research results. One must realize that statistical significance does not equate to clinical significance. Statistical significance addresses only the possibility that the results could have occurred by chance alone, and, as such, is dependent on factors such as sample size. Statistical significance tells you absolutely nothing about the actual magnitude of the clinical differences between groups. These and other related issues will be discussed in greater detail in a future segment of this series specifically focusing on statistics.
      Although the processes described in this paper may seem tedious and laborious (at times they are), they are necessary. Many of the papers you find during this phase of the project will be used to design your research proposal, support a grant application, or aid in manuscript preparation to present your findings. As such, it is useful to take notes as you read the articles, highlighting significant points and major flaws. One helpful method is to number the articles, keeping ordered notes concerning each paper. After completion, summarize your findings, save an electronic copy, and attach a hard copy to the manuscript itself for future reference.

      Conclusions

      Finally, after all of this careful, critical review, you can determine whether enough satisfactory work on the topic has been done to adequately answer your original research question. If not, you have the rationale needed to move ahead with your planned study. You are fast becoming an expert on the subject at hand and have a clear picture of how to begin formulating the actual project. The next step is to refine the research question in light of your new knowledge and select an appropriate research design.

      References

        • Thompson CB
        • Panacek EA
        Basics of research part 1: clinical research and critical care transport: how to get started.
        Air Med J. 2006; 25: 107-111
        • Burns N
        • Grove SK
        The Practice of Nursing Research: Conduct, Critique, & Utilization. 5th ed. W.B. Saunders Company, Philadelphia2005
        • Miller RA
        • Groth T
        • Hasman A
        • Safran C
        • Shortliffe EH
        • Haux R
        • et al.
        On exemplary scientific conduct regarding submission of manuscripts to biomedical informatics journals.
        Methods Inf Med. 2006; 45: 1-3
        • Schroter S
        • Tite L
        • Hutchingss A
        • Black N
        Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors.
        JAMA. 2006; 295: 314-317
        • Smith R
        The trouble with medical journals.
        J R Soc Med. 2006; 99: 115-119
        • Davis EA
        • Thompson CB
        • Panacek EA
        Basics of research part 2: reviewing the literature.
        Air Med J. 1995; 14: 101-105
        • Allen P
        • Jacobs SK
        • Levy JR
        Mapping the literature of nursing: 1996-2000.
        J Med Libr Assoc. 2006; 94: 206-220
        • National Library of Medicine
        Fact Sheet: medical subject headings (MeSH).
        (Accessed May 29, 2006.)
        • Woolard RH
        • Carty K
        • Wirtz P
        • Longabaugh R
        • Nirenberg TD
        • Minugh PA
        • et al.
        Research fundamentals: follow-up of subjects in clinical trials: addressing subject attrition.
        Acad Emerg Med. 2004; 11: 859-866
        • Honest H
        • Khan KS
        Reporting of measures of accuracy in systematic reviews of diagnostic literature.
        BMC Health Services Research. 2002; 2 (Accessed July 7, 2006.): 4