Chapter 2 Basic terminologies: Developing statistical methods in order to draw conclusions about whether a drug is helpful in real-world clinical practice

Medical research seeks to find out whether a treatment works for a disease or condition typically depends on comparing the results of two groups of people – those who get the treatment versus those who do not. The best research makes sure that the people in both groups are very similar (e.g. same age and have had a similar seriousness of the disease for an equal length of time) and that patients don’t get to choose or know whether they got the treatment or not (known as a placebo), until after the study is finished. Unfortunately, this type of research often does not include patients who are the most sick, of older age, or are from a different ethnic group. Also, sometimes there are patients who are unable to continue with the treatment and either take less of the drug or need to drop-out of the study, making it difficult to capture/measure their data. Therefore, it is difficult to accurately analyze the results of the study and be sure if the treatment would work as well on everyone who has the disease. This research project will use a new way to statistically analyze the results of a study to help patients and their doctors know whether a treatment is likely to work for them.

What is ‘treatment effectiveness?’

Let us consider that a drug company is proposing a new drug for curing patients suffering from a particular disease. To decide whether this new drug is actually effective in curing that disease or not, clinical trials are considered a gold standard. These clinical trials are designed carefully to make sure we can find out whether that drug is useful or not under a controlled condition. By control condition, I mean making sure that the trial participants are similar to each other in terms of age, sex and various other patient characteristics – controlling the conditions within our power. Ensuring such a control condition will enable statisticians to meaningfully compare the two groups:

• the group who took the drug (treated group), versus
• the group that was not given the drug (control group).

If people from these two groups are mostly similar, but after the clinical trial end date, if we see there is a difference in cure /survival rate in these two groups, then we can attribute such difference as an ‘effect’ due to the drug only. We call this difference a treatment effect.

How can careful design of a standard clinical trial help us obtain a good understanding of how good a drug is?

In these clinical trials, patients are assigned to different treatment groups at random (e.g., via coin toss). This randomization is a powerful tool that enables us to be free from bias. As the process of treatment assignment involves randomization, there is no reason for us to selectively assign older or sicker patients to the treated group, or selectively assign younger or healthier patients to the control group, thus creating bias. Statistically, it can be shown that when we have enough sample size, selecting patients by the process of randomization will eradicate most sources of bias.

However, if we are not careful in patient selection, it is possible that the two groups could be somewhat different, particularly in small sample sizes. For example, consider a scenario where all the people from the treated group are much older than those in the control group. In that case, if we see that the cure /survival rate in these two groups are different, it is not clear how much of this difference is due to the drug, how much of it is due to age.

Why do we need to be very careful about which patients are selected in a standard clinical trial?

As running clinical trials are extremely expensive, and sometimes risky for frail patients (e.g., severely ill or patients with much older age), patients who are entered into the trial are highly selective, and the eligibility criteria for entering in the clinical trial is very restrictive (e.g., not at high risk, not suffering from any other related diseases, or not taking any medication that interacts with the drug under consideration). Such well-controlled clinical trials will be able to show whether there was any effect of that drug in ideal conditions. That means, the design of clinical trials are purposefully perfected for finding out how well the treatment works.

Where do the regular clinical trials fail?

Unfortunately, such a purist approach has consequences. Let us consider the example of multiple sclerosis (MS). Many of the drugs that are popularly used to treat MS were approved based on evidence from clinical trials that excluded patients over the age of 50. However, only considering British Columbia alone, about one-third of the older population (over 50 years of age) are prescribed these exact drugs in clinical practice today. Even though we do not know precisely whether these drugs are effective for people over 50 years of age, doctors continue to prescribe these patients these expensive drugs without much evidence to rely on.

What is a clinical trial protocol?

Trial protocols are “documents,” created by researcher (who conducts the clinical trials), that outlines how the trial will be conducted. These documents carefully describe clinical trial objectives (e.g., which two treatments will be compared), trial design and organization (e.g., how long the studies will continue, how long the patients should continue with the treatment or placebo, under what condition or side-effect they should interrupt the treatment), statistical considerations (e.g., how many patients should be recruited, which statistical techniques will be used). The trialists then try to follow the protocol document during the trial implementation as close as possible.

What is a real-world pragmatic clinical trial? Is it more relevant from a patient perspective?

‘Pragmatic trials’ include a broader range of subjects compared to a regular clinical trial. . These are the subjects who already receive the treatment in everyday clinical practice in a real-world usual care scenario. Therefore, this population, selected for a pragmatic trial, is much more representative of the real-world than the subjects selected for a regular clinical trial. Consequently, the conclusions from these pragmatic trials are more suitable and relevant for health care decision-makers and a greater number of patients.

Challenges with Pragmatic trials in estimating treatment effect: statistical analysis of these data is not so simple anymore.

Similar to regular clinical trials, subjects are randomized to different treatment groups at the beginning of the trial (known as the baseline period). Unfortunately, pragmatic trials offer different sets of challenges. As in the usual care scenario (under pragmatic trials), patients can deviate from the protocol, e.g., they may not be adherent to the treatment they were assigned at the beginning of the trial, they may switch treatment, or discontinue as people do in a real-world setting. However, to make the analysis of the data simple, treatment effects are often calculated based on ‘intent-to-treat’ methods. That is, the ‘intent-to-treat’ analysis is done based on which treatment groups (treated or control) patients were assigned to at the beginning of the trial. The analysis does not take into account whether patients actually receive the treatment or not.

What is the effect of the treatment on the patients who ‘actually took the drug?’

One can imagine that, at the end of a trial, the effect of a precisely same treatment will not be identical for the person who took treatment during the entire period of the clinical trial (e.g., 2 years), compared to the person who took treatment only half that time (e.g., 1 year). Statistically, it has been shown that the ‘intent-to-treat’ estimate is influenced by the degree of adherence to the treatment. The estimate of the treatment effect will differ if all patients adhered to the treatment protocol/instruction versus if half of the patients stopped taking the treatment half-way through. This simple method provides an average estimate lumping together the people who may or may not have taken the treatment thoroughly. Such averaged-out results are unlikely to be useful for any patient in deciding whether or not to take the medication. These results will not accurately reflect the true benefit of the treatment for a given patient who would completely take the treatment as instructed. Patients and their healthcare professionals would be likely more interested in results that are more clinically relevant to them. They would like to know, if a patient takes a given treatment, what is his/her chance of getting cured. A patient is unlikely to be interested in knowing the effect of that treatment in someone who has different characteristics (age, sex etc.) than him/her.

Why analysis strategies used for standard clinical trials fail analyzing pragmatic trial data?

It is clear from the discussion above that, in analyzing the treatment effect, we need to consider this non-adherence to the treatment. Based on this observation, a few other estimation methods were proposed (known as per-protocol and as-treated methods). However, the problem with these methods is that, it is often the case that the randomization principle will be violated when implementing these methods, which is the basis for many clinical trials. As a result of using these simplistic approaches, the results obtained from these analyses are often biased and misleading. On one side, intent-to-treat methods are simple but give potentially wrong results for a patient about the effect of the treatment. Per-protocol methods take into account some aspects of non-adherence, but the reason for treatment discontinuation is not taken into account in the analysis. Hence, the results from this method are also misleading. If a person had to stop taking the treatment due to a side-effect or due to the patient’s perceived impact of the treatment, proper statistical techniques should be used to address such patterns accordingly.

‘Causal inference’ methods can address some of the above challenges

Fortunately, there has been a large number of methodological and statistical research done for the scenario where randomization may not be present. These research try to carefully identify ‘cause’ of a disease outcome. For example, if a person is getting cured from a disease, whether that is a result of taking the drug or some other reason. Such research is often known as careful analysis of observational data (without randomization) using epidemiologic considerations. Epidemiology is the study of distribution and determinants of disease in populations, epidemiologic considerations mean understanding possible bias sources so that we can remove that bias, and the resulting conclusions from the analysis are meaningful and interpretable (e.g., free from bias, and hence our inference is based on cause and effect). For our context, accounting for bias sources could mean, while trying to decide whether a drug works or not, we think hard about what are the sources of bias e.g., some patients who are supposed to take a prescribed drug, did they actually took the drug. If they did not take the prescribed drugs, then the results would be biased, and we need to apply appropriate methodological and statistical techniques to remove such source of bias. Many of the methods proposed in this area of research (‘causal inference’ methods) have the potential to be applied to the area of pragmatic clinical trials, e.g., particularly, to deal with non-adherence. In the following part we will discuss them in further details.