Use of real-world data for measuring treatment effectiveness for target populations – Healthcare Economist

Randomized controlled trials are the gold standard for evaluating treatment effectiveness, but real-world effectiveness may vary. One reason for this is that clinical trials often have stricter inclusion criteria than the population treated. Policymakers, payers, and clinicians may wonder how well results translate from the smaller clinical trial population to the real-world “target” population.

This is the question from an article by Lugo-Palacios et al. (2024) intends to respond. The goal of their study is to determine which second-line treatment for type 2 diabetes is most effective in the real world. To do this, the authors estimate the average treatment effect (ATE) and the conditional average treatment effect (CATE) for the use of dipeptidyl peptidase 4 inhibitors (DPP4i) and sulfonylureas (SU) as “add-on” therapies to the metformin for treatment. of patients with type 2 diabetes in England. The primary endpoint of interest was glycemic control. One challenge is that published RCT reports do not have a consensus recommendation; some find greater improvement with SU and others with DPP4i. As mentioned above, one problem is that RCTs evaluating these treatments often exclude patients with very poor glycemic control, and therefore it is unclear to what extent different types of real-world patients would benefit. of each treatment.

The study approach identified subpopulations within the target population into two groups: those who met the eligibility criteria of a published RCT (“eligible RCT”) and those who did not meet the eligibility criteria (“ineligible RCT”). The authors compare the ATE for “eligible RCTs” to RCT with the same eligibility criteria (the “RCT benchmark”) to examine how well real-world data mimic RCT data. The authors then compared CATEs for the overall target population (i.e., “RCT-eligible” and “RCT-ineligible” groups). CATEs were estimated separately by age, ethnicity, baseline HbA1c, and body mass index (BMI). Covariates used in the analysis included demographic and clinical factors (i.e., baseline HbA1c, systolic blood pressure (SBP), diastolic blood pressure (DBP), estimated glomerular filtration rate (eGFR), and BMI).

The econometric approach was to use local instrumental variables (LIV). The instrument used was

…the tendency of clinical commissioning groups (CCGs) to prescribe (TTP)iDPP4 as second-line treatment. During the study period, general practitioners (GPs) worked within a CCG which informed health funding decisions for their respective geographical region. For example, some CCGs tended to recommend – to their affiliated GPs – the prescription of DPP4i or SU.

Using this instrument, the authors estimated the LIV as follows:

…the first-stage models estimated the probability that each person would be prescribed DDP4i given their baseline covariates and their GCC TTP. The second-stage outcome models then included the predicted probabilities from the first-stage models (propensity score), covariates, and their interactions. Probit regression models were used to estimate the initial propensity score (first stage), while generalized linear models were applied to the outcome data, choosing the most appropriate family (Gaussian) and link function (identity) based on the root of the mean square error, with the Hosmer‐Lemeshow and Pregibon tests were also used to check the fit and suitability of the model.

Using this approach, the authors found the following:

The IV was the tendency of clinical commissioning groups (CCGs) to prescribe (TTP) iDPP4 as second-line treatment. During the study period, general practitioners (GPs) worked within a CCG which informed health funding decisions for their respective geographical region. For example, some CCGs tended to recommend – to their affiliated GPs – the prescription of iDPP4i or SU as a second-line treatment.

The authors use this approach and find that:

The estimated ATEs for the “RCT-eligible” population are similar to those from a published RCT. The estimated CATEs go in the same direction for the subpopulations included and excluded from the RCT, but differ in magnitude. The variation in the estimated individual treatment effects is greater in the larger sample of people who do not meet the RCT inclusion criteria than in those who do.

The graphs show overall results for eligible and ineligible RCTs as well as specific subgroups of interest.

Learning point

What are the 4 conditions that a valid instrument must meet? The authors describe them as follows.

First, the instrument must predict the prescribed treatment… Second, the instrument must be independent of the unmeasured covariates that predict the outcomes of interest, which can be partially assessed through their relationship with the measured covariates… Third, the instrument should have an effect on outcomes only through the treatment received… Fourth, we assume that the average treatment choice should monotonically increase or decrease with the level of the IV.