Balance Diagnostics
Video content
Slides
Summary of topics discussed
Definition
The Standardized Mean Difference (SMD) is a statistical measure used to quantify the size of the difference between two groups (binary exposure, but generalization is possible). It’s often used in meta-analyses and systematic reviews to summarize the results across various studies, especially when different scales are used to measure an outcome.
Balance measures
SMD, a continuous balance measure, serve as a pivotal and broadly utilized statistical measure in propensity score research, offering a scale-free metric that enables the comparison of group differences and balance. Balance is aimed towards the sample at hand, and SMD is a sample characteristics. For further insights and practical applications, refer to the related lab materials). We should note that PS model building is an iterative approach. When the balance of measured baseline covariates between exposed and unexposed groups (such as SMD) is satisfactory in the matched sample, we cease further refinement of our PS model building.
- The use of statistical significance testing to compare the baseline characteristics of treated and untreated participants in propensity score matched samples is generally discouraged.
- Employing diagnostics that compare the distribution of the propensity score between treated and untreated participants is also advised against. Research underscores that such comparisons fail to provide valuable information regarding the adequacy of the propensity score model’s specification.
- The c-statistic, which is related to the Receiver Operating Characteristic (ROC) curve, of the propensity score model is not deemed useful in this context. It does not furnish information pertinent to determining the adequacy of the model specification.
Application
SMD is useful in comparing the difference between groups when the variable is measured in different units across studies. An SMD of 0 indicates no difference (suggesting balance) between the groups, while its absolute value provides the size of the imbalance (e.g., |SMD| = 0.2: small imbalance, 0.5: medium imbalance, 0.8: large imbalance). While RCTs are designed to inherently balance covariates between groups through randomization, observational studies often require additional statistical methods (e.g., matching or weighting) to achieve and assess balance, with SMD being a valuable metric in this context.
References (Optional)
- Franklin, Jessica M., Jeremy A. Rassen, Diana Ackermann, Dorothee B. Bartels, and Sebastian Schneeweiss. 2014. Metrics for Covariate Balance in Cohort Studies of Causal Effects.
Related content was discussed in previous course materials.