Contact us

  • Liese Kelly
    .(JavaScript must be enabled to view this email address)
  • Alan Clarke
    .(JavaScript must be enabled to view this email address)

The Bridge Consultancy
Richmond Bridge House
419 Richmond Road

020 8334 5522


Send us your CV

Latest tweets

  • 3:36, 16 Mar
    ‘Smell-o-vision’ brings The Gruffalo to life for visually impaired…
  • 11:17, 11 May
  • 8:25, 11 Sep
    Well done guys! A brilliant achievement for a brilliant cause.…
  • Share this page

    A few extra pointers

    Real-World Evidence; What is it and what can it tell us?

    Real-World Evidence — What Is It and What Can It Tell Us? Rachel E. Sherman, M.D., M.P.H., Steven A. Anderson, Ph.D., M.P.P., Gerald J. Dal Pan, M.D., M.H.S., Gerry W. Gray, Ph.D., Thomas Gross, M.D., M.P.H., Nina L. Hunter, Ph.D., Lisa LaVange, Ph.D., Danica Marinac‑Dabic, M.D., Ph.D., Peter W. Marks, M.D., Ph.D., Melissa A. Robb, B.S.N., M.S., Jeffrey Shuren, M.D., J.D., Robert Temple, M.D., Janet Woodcock, M.D., Lilly Q. Yue, Ph.D., and Robert M. Califf, M.D. The term “real-world evidence” is widely used by those who develop medical products or who study, deliver, or pay for health care, but its specific meaning is elusive. We believe it refers to information on health care that is derived from multiple sources outside typical clinical research settings, including electronic health records (EHRs), claims and billing data, product and disease registries, and data gathered through personal devices and health applications.1,2 Key to understanding the usefulness of real-world evidence is an appreciation of its potential for complementing the knowledge gained from traditional clinical trials, whose well-known limitations make it difficult to generalize findings to larger, more inclusive populations of patients, providers, and health care delivery systems or settings that reflect actual use in practice.3 Real-world evidence can inform therapeutic development, outcomes research, patient care, research on health care systems, quality improvement, safety surveillance, and well-controlled effectiveness studies. Real-world evidence can also provide information on how factors such as clinical setting and provider and health-system characteristics influence treatment effects and outcomes. Importantly, the use of such evidence has the potential to allow researchers to answer these questions efficiently, saving time and money while yielding answers relevant to broader populations of patients than would be possible in a specialized research environment.4,5 As defined above, real-world evidence can be viewed as a means of incorporating diverse types of evidence into information on health care. However, the confluence of large data sets of uncertain quality and provenance, the facile analytic tools that can be used by nonexperts, and a shortage of researchers with adequate methodologic savvy could result in poorly conceived study and analytic designs that generate incorrect or unreliable conclusions. Accordingly, if we are to realize the full promise of such evidence, we must be clear about what it is and how it can be used most effectively, and we must have appropriate expectations about what it can tell us. It is important to distinguish two key dimensions of real-world evidence. The first is the setting in which evidence is generated, which includes the population defined by the data source as well as the specific methods used to collect and curate the data on that population. The second is the methodologic approach used to conduct the surveillance or research. Research Set tings — Tr aditional Trials vs. Real World “Traditional” clinical trials are often conducted with specific populations and in specialized environments that differ from the realities of clinical or home settings. These trials may take measures designed to control variability and to ensure the quality of the data they generate, such as the development of long lists of eligibility criteria, the use of detailed case-report forms that exist separately from ordinary medical records, and the use of intensive monitoring and specialized research personnel to ensure adherence to a well-characterized protocol that defines study procedures and ensures precision in data collection. The clinical trial unquestionably remains a powerful tool for developing scientific evidence about the safety and efficacy of a medical product while informing our understanding of the The New England Journal of Medicine Downloaded from at UNIVERSITAET ULM /KIZ on January 3, 2017. For personal use only. No other uses without permission. Copyright © 2016 Massachusetts Medical Society. All rights reserved. The new england journal o f medicine 2294 n engl j med 375;23 December 8, 2016 biologic mechanisms involved in its therapeutic action. These trials are often needed because they are designed to provide an essential element of the premarket evaluation of a medical product — namely, robust evidence that a treatment may “work.” However, the internal validity attained in these trials is often achieved at the expense of uncertainty about generalizability, especially since the populations enrolled in such studies may differ in significant ways from those seen in practice. In addition, there may be few data on interactions with concomitant illnesses and treatment, and adherence to therapies may be supported by intensive efforts that are infeasible in practice. Moreover, the expense of conducting large traditional trials has been growing steadily for years,6 and recent estimates suggest that the cost trajectory may be steepening,7 without any indication of a commensurate increase in the quantity of evidence produced to support decisions about health care. Given these trends, many trialists, clinical researchers, and medical-product developers have become increasingly interested in expanding and integrating clinical research into more diverse, real-world settings by capitalizing on the exponential growth in access to data from EHRs, claims databases, electronic devices and software applications (or apps), registries embedded in clinical practice, and social media. These sources can provide new insight into states of health and illness. For instance, EHRs, registries, and claims databases contain rich data that are already being gathered in real-world settings at the point of care, personal devices and apps allow continuous monitoring and data capture8 and facilitate shared decision making,9 and data from social media can be used for epidemiologic purposes.10 But these data sources also raise concerns. EHR and claims data are not collected or organized with the goal of supporting research, nor have they typically been optimized for such purposes, and the accuracy and reliability of data gathered by many personal devices and health-related apps are unknown.11,12 Furthermore, the use of any of these sources, including social media, raises important questions about the quality of the data they provide and about privacy. The technological and methodologic challenges presented by these new data sources are the focus of active efforts by researchers. For example, multiple stakeholders, including the Food and Drug Administration (FDA), are working on ways to harmonize data collected from EHRs, claims data, and registries to create a unified system for monitoring the safety and effectiveness of medical devices.13,14 Others, such as the National Institutes of Health (NIH) Collaboratory (an NIH Common Fund initiative devoted to building infrastructure, operational knowledge, and capacity for pragmatic research in the context of health care systems),15 are developing and implementing methods for incorporating data from EHRs and other sources into research. Such efforts include the development of largescale distributed research networks16 and “computable phenotypes” (i.e., conditions or patient characteristics that can be derived from EHRs and claims data without requiring external review or interpretation17) that allow researchers to identify cohorts of interest across multiple data sources.18 Research Methods, Treatment Alloc ation, and the Definition of Real-World Evidence We believe that real-world evidence can be used across a wide spectrum of research, ranging from observational studies to studies that incorporate planned interventions, whether with or without randomization at the point of care. At the same time, however, it is incorrect to contrast the term “real-world evidence” with the use of randomization in a manner that implies that they are disparate or even incompatible concepts. As we adapt the tools and methods of traditional trials to real-world settings, we must consider the components of such trials that are critical to obtaining valid results and minimizing bias.19 Although real-world evidence can be used in multiple research scenarios, the selection of appropriate analytic approaches will be determined by key dimensions of the study design, including the use of prospectively planned interventions and randomization.20 Planned interventions, whether randomized or not, can be used in both the tertiary care and academic environments, where much clinical research is typically performed in association with intensive support and expensive resources. These interventions can also be used in “real-world” settings with less labor-intensive clinical research support and possibly a lesser degree of familiarity with cliniThe New England Journal of Medicine Downloaded from at UNIVERSITAET ULM /KIZ on January 3, 2017. For personal use only. No other uses without permission. Copyright © 2016 Massachusetts Medical Society. All rights reserved. Sounding Board n engl j med 375;23 December 8, 2016 2295 cal research. For this reason, discussions of realworld evidence must be informed by a clear understanding of the methods used, so that the best methods that have been developed and validated can be combined with the most appropriate research settings. In traditional trials, randomization has long been an essential tool for minimizing bias by balancing underlying risk between treatment groups, but it can be just as useful and important in real-world studies. In fact, one of the first major randomized, controlled trials (RCTs) conducted in a real-world setting was the Salk field trial of the polio vaccine, which combined a large component comprising 750,000 children who were randomly assigned to receive vaccine or placebo (control group) with an even larger nonrandomized “observed control” group of 1 million children, all of whom received the vaccine.21 A contemporary version of a large, simple trial performed in a real-world setting is the “Aspirin Study,” also known as ADAPTABLE (Aspirin Dosing: A Patient-centric Trial Assessing Benefits and Long-Term Effectiveness), which is being conducted by the National Patient-Centered Clinical Research Network. In this trial, 20,000 participants are being randomly assigned to one of two commonly used doses of aspirin in order to ascertain which of these two dose regimens is better for the secondary prevention of cardiovascular disease.22 There is extensive literature on pragmatic RCTs designed to inform decision making at the individual and the population level.23 Many of the NIH Collaboratory’s demonstration projects involved innovative pilot approaches to performing pragmatic research within health systems.24 Cluster randomization, which is particularly useful for evaluating interventions at the level of health systems, practices, or hospitals, was used for most of these projects.25 In addition to its application in interventional studies, real-world evidence is also valuable in observational settings, where it is used to generate hypotheses for prospective trials,26 assess the generalizability of findings from interventional trials (including RCTs),27 conduct safety surveillance of medical products,28 examine changes in patterns of therapeutic use, and measure and implement quality in health care delivery.1 However, much of the current excitement about realworld evidence stems from the hope that access to sources of emerging data of adequate quality will, when paired with the development of more robust methods, allow greater use of observational treatment comparisons in drawing causal inferences about the treatment effects of medical products. Although observational studies are an essential tool for clinical epidemiologic investigations, quality improvement, and safety surveillance, their findings require judicious evaluation when used to assess treatment effects.29 These limitations are particularly problematic when an observational study is used to evaluate the effectiveness of a medical product and the expected or observed effect is relatively small. When this is the case, it can be difficult to be confident that the effect is not due largely or wholly to confounding factors. This problem, compounded by the fact that observational studies often leverage existing rather than prospectively collected data (e.g., as part of a disease or product registry with well-established quality standards), can add to the uncertainty regarding findings and limit the usefulness of such data. Awareness of the limitations of source data and analytic approaches1 is fueling concern that when the term “real-world evidence” is used in such contexts, the allure of analyzing existing data may lead to flawed conclusions. This concern is especially salient in light of the growing proliferation of precision molecular medicine and treatments for rare diseases, many of which are anticipated to undergo review in accelerated approval programs. In such circumstances, realworld evidence will become an increasingly critical element in expediting the availability of data needed to confirm clinical benefit and value, because products will necessarily receive initial approval in an atmosphere in which there is greater uncertainty with regard to clinical outcomes. Although access to real-world data adds important dimensions to the assessment of therapies and important progress is being made in the methodologic arena,30 these factors do not yet suffice to fully overcome the fundamental issues of confounding, data quality, and bias,31 unless other, specific countervailing features of the evaluation are relevant. For example, prospective registries or singlegroup trials with planned external controls and high-quality data collection have been accepted for regulatory purposes in the evaluation of medical devices (e.g., a ventricular-assist system The New England Journal of Medicine Downloaded from at UNIVERSITAET ULM /KIZ on January 3, 2017. For personal use only. No other uses without permission. Copyright © 2016 Massachusetts Medical Society. All rights reserved. The new england journal o f medicine 2296 n engl j med 375;23 December 8, 2016 that used propensity-score–matched controls from the Interagency Registry for Mechanically Assisted Circulatory Support32). However, because medical devices are typically developed in an iterative fashion, building on earlier designs and incorporating refinements throughout the product life cycle, substantial knowledge of the effect of confounding factors is often available a priori. This availability in turn facilitates the evaluation of observed treatment effects, as exemplified by the use of data from the Transcatheter Valve Therapy Registry for postmarketing regulatory purposes, including labeling revisions.33 Thus, although we are optimistic about longterm prospects for the evolution of mature, robust methodologic approaches to the incorporation of real-world evidence into therapeutic development and evaluation given the intensive efforts now under way, caution is still needed, and expectations of “quick wins” resulting from the use of such evidence should be tempered accordingly. Specifically, other analytic methodologies with varying levels of evidentiary requirements, such as historical controls or study designs with an open-label phase in which all patients receive the investigational product, fall within the spectrum of potentially useful approaches that will require careful consideration before they can be appropriately applied to answer important questions about the effects of treatment with medical products in real-world settings, including issues involving latent or rare outcomes and treatments for rare diseases. To this end, the FDA is committed to robust policy development under the proposed reauthorization of the Prescription Drug User Fee Act VI (user-fee program) for drugs and biologic products.34 This commitment includes convening public workshops involving participants on all bands of the research spectrum — from patients to providers to sponsors — to gather input on the use of real-world evidence in regulatory decision making. With this information, the agency will initiate activities to address key concerns and publish draft guidance on how such evidence can be used to assess safety and effectiveness in both premarketing and postmarketing regulatory requirements. Complementary efforts are included in Medical Device User Fee Amendments IV for devices.35 Conclusions We believe that when the term “real-world evidence” is used, the primary attribute that distinguishes it from other kinds of evidence is related to the context in which the evidence is gathered — in other words, in clinical care and home or community settings as opposed to researchintensive or academic environments. Most important, the distinction should not be based on the presence or absence of a planned intervention or the use of randomization. Real-world research and the concepts of a planned intervention and randomization are entirely compatible. Indeed, one of the most important advances in clinical trial methodology may be the broadening of the application of randomization outside more typical venues for clinical trials, such as academic research centers. But in order to gain collective confidence in the appropriate uses of this array of methods across disparate settings, we must first be clear about our terminology and its application. Disclosure forms provided by the authors are available with the full text of this article at We thank Jonathan McCall, M.S. (Duke Clinical Research Institute, Durham, NC), for editorial assistance with an earlier version of the manuscript.