What study type should I choose? | How to choose a data source or sources? | What are the advantages of data linkage? | Should I incorporate Artificial Intelligence (AI) or Machine Learning (ML)? | How to minimize bias and confounding? | How to evaluate the quality of a RWE study?

What study type should I choose?

Most RWE studies are observational rather than interventional and may be descriptive or analytic. Descriptive studies focus on a single group with no comparison group, and may examine, for example, the epidemiology of a disease, the effectiveness or safety of a particular therapy, or the impact of a treatment on quality of life.

Analytic studies do incorporate a comparison group and may either track patients from initial exposure to a specified outcome (cohort study), identify patients with a particular outcome and track them back in time to an exposure of interest (case-control study), or look at both exposure and outcome at a single point in time (cross-sectional study). Case-control and cross-sectional studies are performed retrospectively utilizing already existing data. Cohort studies may be performed retrospectively or by prospectively following identified cohorts.

Each type of study design has both strengths and limitations. The study design selected should both be feasible and able to address the research question posed.

How to choose a data source or sources?

Primary Real World Data (RWD) is data that is collected specifically for the purpose of a study, and is collected prospectively or sometimes cross-sectionally. Secondary RWD is collected retrospectively from a source whose data was collected for a different purpose, such as patient health records. RWD may include both structured data, such as laboratory results, and unstructured data, such as physician notes.

The data source selected should be “fit for purpose”. That is, in addition to assurances of data quality and integrity, it must include the patient population(s) relevant to the research question, contain the data measurements needed to answer the research question collected over a sufficiently long period, and should be of sufficient size to enable a meaningful analysis.

What are the advantages of data linkage?

Certain insights may not be possible to derive from a single data source. Linking records across datasets, for example across outpatient and inpatient settings, can give a more complete picture of a patient’s health, treatment and outcomes that may help in answering complex research questions. There are several challenges to linking datasets, for example the need for data standardization, interoperable systems, and maintenance of patient privacy.

Should I incorporate Artificial Intelligence (AI) or Machine Learning (ML)?

AI is a general concept referring to building systems that mimic human cognition and can streamline and improve the performance of time-consuming or repetitive tasks. ML is the subset of AI that enables a system to identify patterns and learn insights from large amounts of data without being specifically programmed to do so.

There are certain contexts in which ML can provide significant value to a RWE study – in particular, when the aim is to produce patient-level predictions and when there are many variables involved. It is important to be aware, however, that depending on the algorithm or model used in a ML prediction it may not be possible to extract a detailed understanding of how the ML system arrived at its prediction.

How to minimize bias and confounding?

Bias and confounding in RWE studies can be reduced both prior to conducting the study, by study design decisions, as well as after the study has been conducted, through statistical analysis.

Study design:

Restriction – Inclusion/exclusion criteria may be applied to eliminate characteristics that may be potential confounders – such as age, certain pre-existing conditions or medications.

Matching – Simple characteristics like age and sex may be used to select controls who resemble the exposed population.

Propensity score (PS) methods – Unlike RCT, in RWE studies treatment decisions or assignments may be influenced by factors that can also affect the treatment outcome. PS is the probability that an individual will receive a particular treatment given a set of important covariates. Several methods utilize the PS to reduce confounding by extracting treated and control populations that are more comparable with respect to the important identified covariates. In PS matching (PSM), the treated population is matched with controls having a similar PS, and patients who cannot be matched are excluded. An alternate approach is to use PS Trimming, which excludes individuals with scores at both tail ends of the PS distribution. An additional method, IPTW (Inverse Probability of Treatment Weighting), enables all subjects to be included by creating a pseudo-population of treated and control subjects that simulates a reference population in which the identified covariates and treatments assigned are mutually independent. It should be noted that in all of these methods, confounding can be controlled only for the covariates identified as important.

Analysis:

Imputation – Imputation is an approach to reducing bias due to missing data. Various methods can be used to produce single estimates of missing values (Single Imputation); or Multiple Imputation can be used to produce and combine results from several different generated imputed data sets.

Regression – Regression is a statistical method to control for unbalanced differences between groups by mathematically describing the relationship between a dependent variable, such as outcome, and one or more independent variables, such as treatment. In this way, it can estimate the significance of association between the independent and dependent variables while controlling for known, unbalanced confounders. Three regression models are commonly used in RWE analysis. Linear regression is used to explain a continuous outcome.; logistic regression models the probability of an event occurring for a dichotomous outcome; Cox regression is used for time-to-event outcome analyses.

Stratification – Results are grouped according to potential confounders, such as age or sex, and then treatment effects for each group or stratum are compared to determine if the putative confounder in fact exerts an effect.

How to evaluate the quality of a RWE study?

In order to confidently apply the findings of a RWE study, it is important that the study addresses the question(s) of interest, and that the quality of the contributing RWD, the data analysis performed and the interpretation of results obtained be robust.

PICOT is a tool that helps the researcher formulate the question to be investigated. The acronym directs the user to define the question in terms of Population, Intervention, Comparison or Control, Outcome and Time period.

In addition, resources have been developed both to assist researchers in designing and reporting RWE studies, and to evaluate the quality of these studies.

Design and reporting:

STROBE describes the key elements that should be included in an observational study report, focusing on cohort, case-control and cross-sectional study designs, and is a useful checklist not only for authors, but also for readers and reviewers for evaluating an article’s quality.

STaRT-RWE is a more recently developed tool that includes a series of template tables and a design diagram whose completion unambiguously describes the RWE methods utilized.

Quality Evaluation:

The GRACE checklist includes a series of Yes/No questions relating to study data and methods designed to help evaluate the quality and usefulness of observational studies comparing 2 or more treatments.

ISPOR (The International Society for Pharmacoeconomics and Outcomes Research) developed a questionnaire to assist in evaluating the quality of observational studies. The questions are grouped according to relevance and credibility, where credibility includes a consideration of study design, data, analyses, reporting, interpretation and conflicts of interest. Certain assessments constitute fatal flaws which identify a need to use caution in deciding whether to use the study results in decision-making.

ROBINS-I specifically addresses the risk of bias in non-randomized interventional studies. This tool addresses seven domains in which bias may occur and evaluates the risk of bias by considering how well the non-randomized study imitates a hypothetical pragmatic randomized trial (which might be neither feasible nor ethical) asking the same research question. It asks questions relating to each domain which help the user decide the level of risk for each domain and overall, and can be used in designing a study as well as in evaluating journal submissions and publications.