User login
Effect modification: An important, but often underappreciated, statistical concept
In previous articles, we explored the meaning of two statistical concepts: that of the P value1 and that of confounding.2 In today’s article, we will focus on another important, though often underrecognized, statistical concept: that of effect modification (sometimes also called “interaction,” although many authors draw a distinction between these terms).3-5 In brief, effect modification is the recognition that the relationship between two variables can be different based on the values of a third variable. Let me illustrate with an example. Suppose that you give a weight-loss drug to a group of people. After a predetermined time, say 3 months, you weigh them to measure the effects of the drug, and you record that on average each individual lost about 10 pounds. Suppose, however, that you are curious to see the effect of the drug in men and women separately. You perform an analysis stratified by sex, and you notice that men lost on average only about 1 pound each, whereas women lost on average about 30 pounds each. What do these results mean? It seems that the effect of the drug on weight loss is different depending on the value of a third variable, namely sex. That is, in this case, sex acts as an effect modifier. Based on this analysis, we may conclude that the drug has real effects on women but not on men.
Before reaching this conclusion, however, it is appropriate to ask whether we have made a mistake. The first obvious thing is to make sure that we have not made a mistake in our collection of data. Once we have excluded the presence of structural bias in our dataset, how can we ascertain that these results, which at eyeballing seem so different, have not been so as a result of chance? Fortunately, we do not have to guess. There is a way to formally test for this hypothesis. If sex is truly an effect modifier, then we can perform what is called in statistical terms an “interaction term” between the exposure (in this case the drug) and the potential effect modifier (in this case sex) in a multivariable model that includes both as exposures. If the P value for that interaction term is less than .05, then the interaction term is statistically significant, and therefore the variable (in this case sex) is confirmed to be an effect modifier. Hence, the results are not due to chance, and the different effects in men and women are plausibly attributable to different biological responses to the medication.
Difference between confounding and effect modification
At this point someone might ask, “What then is the difference between confounding and effect modification? In both cases, we do stratified analysis of the relationship between an exposure (in this case the weight-loss drug) and an outcome (in this case weight loss) based on strata of a third variable (in this case sex).” The difference is fundamental. Confounding, as we explored in a previous article,2 is something that we would like to get rid of. It is the effect of an outside variable on both the exposure and outcome that does not allow us to properly evaluate the real relationship between the two. As such, we try to adjust for this variable, so that its effect is eliminated and we can only observe the relationship between the exposure and the outcome. Effect modification, on the other side, is not something we would like to get rid of. On the contrary, effect modification is part of what we would like to explore and describe because it is part of the biological mechanism that explains the real relationship between the exposure and the outcome. In the example above, if effect modification by sex is confirmed, it implies that there is something in female biology that is not found in male biology (or vice-versa) that makes their response to the medication different and therefore something of interest to study further. Thus, differently from confounding, effect modification is part of the objective reality of the world which we would like to explore and evaluate.
Several questions then arise. How can we know whether a variable is a confounder, an effect modifier, or both? As a general rule of thumb, a confounder would be a variable which, when a stratified analysis is done (or when added to a multivariable model), will change the relationship between the exposure and outcome by 10% or more. However, the relationship between the exposure and the outcome in both strata will be similar. In the example above, it would mean that if sex was only a confounder, then stratifying by sex would show roughly a similar change in the effect (say 9 kg for men and 11 kg for women). An effect modifier on the other hand is one which, when a stratified analysis is done, the association between the exposure and the outcome is very different in the two strata, as illustrated in the example above (for simplicity I am only considering two-level effect modifiers in this article).
Can a variable be both a confounder and an effect modifier? Yes, that is possible. What can be done in this case? The most common approach is to behave the same as when only effect modification is present, namely to show with the interaction term that effect modification exists and present the results between the exposure and outcome separately by the level of the effect modifier (in this example, it means that we need to describe the effects of the weight-loss drug separately for men and women). The stratified analysis/presentation will, by definition, take care of confounding as well.2
Should we always look for effect modification? Not necessarily. As a general rule, we need to test for effect modification only if there is some biological rationale that would compel us to do so, and testing should be hypothesis-driven. A common mistake that some authors make is to perform too many interaction tests and then describe as “positive findings” any test for which P value happens to be less than .05. However, as we pointed out in the article on P value,1 if we perform multiple tests, this increases the probability of false positives, and therefore the probability of spurious findings. Thus, effect modification analysis (with interaction terms) should, generally speaking, be performed with a biological rationale and/or be hypothesis driven.
Conclusion
Effect modification is an essential statistical concept that describes an underlying biological reality in which the association between an exposure and an outcome is different based on the values of a third variable. Differently from confounding, which clouds the association between exposure and outcome, and therefore is something that we try to get rid of, effect modification serves to bring to light a more proper understanding of the biological reality underlying the true association between an exposure and an outcome and as such is something that needs to be explored and described.
Dr. Jovani is assistant professor of medicine, therapeutic endoscopy, digestive diseases, and nutrition at the University of Kentucky Albert B. Chandler Hospital, Lexington. He has no conflicts of interest.
References
1. “The P value: What to make of it? A simple guide for the uninitiated.” GI & Hepatology News. 2019 Sep 23. www.mdedge.com/gihepnews/article/208601/mixed-topics/p-value-what-make-it-simple-guide-uninitiated
2. “Evaluating a paper: Take care not to be confounded.” GI & Hepatology News. 2020 Sep 18. www.mdedge.com/gihepnews/article/228765/mixed-topics/evaluating-paper-take-care-not-be-confounded
3. Corraini P et al. Clin Epidemiol. 2017;9:331-8. doi: 10.2147/CLEP.S162236.
4. VanderWeele TJ. Epidemiology. 2009 Nov;20(6):863-71. doi: 10.1097/EDE.0b013e3181ba333c.
5. Shahar E and Shahar DJ. Epidemiology. 2010 Jul;21(4):587. doi: 10.1097/EDE.0b013e3181e0995c. Author reply 587-8.
In previous articles, we explored the meaning of two statistical concepts: that of the P value1 and that of confounding.2 In today’s article, we will focus on another important, though often underrecognized, statistical concept: that of effect modification (sometimes also called “interaction,” although many authors draw a distinction between these terms).3-5 In brief, effect modification is the recognition that the relationship between two variables can be different based on the values of a third variable. Let me illustrate with an example. Suppose that you give a weight-loss drug to a group of people. After a predetermined time, say 3 months, you weigh them to measure the effects of the drug, and you record that on average each individual lost about 10 pounds. Suppose, however, that you are curious to see the effect of the drug in men and women separately. You perform an analysis stratified by sex, and you notice that men lost on average only about 1 pound each, whereas women lost on average about 30 pounds each. What do these results mean? It seems that the effect of the drug on weight loss is different depending on the value of a third variable, namely sex. That is, in this case, sex acts as an effect modifier. Based on this analysis, we may conclude that the drug has real effects on women but not on men.
Before reaching this conclusion, however, it is appropriate to ask whether we have made a mistake. The first obvious thing is to make sure that we have not made a mistake in our collection of data. Once we have excluded the presence of structural bias in our dataset, how can we ascertain that these results, which at eyeballing seem so different, have not been so as a result of chance? Fortunately, we do not have to guess. There is a way to formally test for this hypothesis. If sex is truly an effect modifier, then we can perform what is called in statistical terms an “interaction term” between the exposure (in this case the drug) and the potential effect modifier (in this case sex) in a multivariable model that includes both as exposures. If the P value for that interaction term is less than .05, then the interaction term is statistically significant, and therefore the variable (in this case sex) is confirmed to be an effect modifier. Hence, the results are not due to chance, and the different effects in men and women are plausibly attributable to different biological responses to the medication.
Difference between confounding and effect modification
At this point someone might ask, “What then is the difference between confounding and effect modification? In both cases, we do stratified analysis of the relationship between an exposure (in this case the weight-loss drug) and an outcome (in this case weight loss) based on strata of a third variable (in this case sex).” The difference is fundamental. Confounding, as we explored in a previous article,2 is something that we would like to get rid of. It is the effect of an outside variable on both the exposure and outcome that does not allow us to properly evaluate the real relationship between the two. As such, we try to adjust for this variable, so that its effect is eliminated and we can only observe the relationship between the exposure and the outcome. Effect modification, on the other side, is not something we would like to get rid of. On the contrary, effect modification is part of what we would like to explore and describe because it is part of the biological mechanism that explains the real relationship between the exposure and the outcome. In the example above, if effect modification by sex is confirmed, it implies that there is something in female biology that is not found in male biology (or vice-versa) that makes their response to the medication different and therefore something of interest to study further. Thus, differently from confounding, effect modification is part of the objective reality of the world which we would like to explore and evaluate.
Several questions then arise. How can we know whether a variable is a confounder, an effect modifier, or both? As a general rule of thumb, a confounder would be a variable which, when a stratified analysis is done (or when added to a multivariable model), will change the relationship between the exposure and outcome by 10% or more. However, the relationship between the exposure and the outcome in both strata will be similar. In the example above, it would mean that if sex was only a confounder, then stratifying by sex would show roughly a similar change in the effect (say 9 kg for men and 11 kg for women). An effect modifier on the other hand is one which, when a stratified analysis is done, the association between the exposure and the outcome is very different in the two strata, as illustrated in the example above (for simplicity I am only considering two-level effect modifiers in this article).
Can a variable be both a confounder and an effect modifier? Yes, that is possible. What can be done in this case? The most common approach is to behave the same as when only effect modification is present, namely to show with the interaction term that effect modification exists and present the results between the exposure and outcome separately by the level of the effect modifier (in this example, it means that we need to describe the effects of the weight-loss drug separately for men and women). The stratified analysis/presentation will, by definition, take care of confounding as well.2
Should we always look for effect modification? Not necessarily. As a general rule, we need to test for effect modification only if there is some biological rationale that would compel us to do so, and testing should be hypothesis-driven. A common mistake that some authors make is to perform too many interaction tests and then describe as “positive findings” any test for which P value happens to be less than .05. However, as we pointed out in the article on P value,1 if we perform multiple tests, this increases the probability of false positives, and therefore the probability of spurious findings. Thus, effect modification analysis (with interaction terms) should, generally speaking, be performed with a biological rationale and/or be hypothesis driven.
Conclusion
Effect modification is an essential statistical concept that describes an underlying biological reality in which the association between an exposure and an outcome is different based on the values of a third variable. Differently from confounding, which clouds the association between exposure and outcome, and therefore is something that we try to get rid of, effect modification serves to bring to light a more proper understanding of the biological reality underlying the true association between an exposure and an outcome and as such is something that needs to be explored and described.
Dr. Jovani is assistant professor of medicine, therapeutic endoscopy, digestive diseases, and nutrition at the University of Kentucky Albert B. Chandler Hospital, Lexington. He has no conflicts of interest.
References
1. “The P value: What to make of it? A simple guide for the uninitiated.” GI & Hepatology News. 2019 Sep 23. www.mdedge.com/gihepnews/article/208601/mixed-topics/p-value-what-make-it-simple-guide-uninitiated
2. “Evaluating a paper: Take care not to be confounded.” GI & Hepatology News. 2020 Sep 18. www.mdedge.com/gihepnews/article/228765/mixed-topics/evaluating-paper-take-care-not-be-confounded
3. Corraini P et al. Clin Epidemiol. 2017;9:331-8. doi: 10.2147/CLEP.S162236.
4. VanderWeele TJ. Epidemiology. 2009 Nov;20(6):863-71. doi: 10.1097/EDE.0b013e3181ba333c.
5. Shahar E and Shahar DJ. Epidemiology. 2010 Jul;21(4):587. doi: 10.1097/EDE.0b013e3181e0995c. Author reply 587-8.
In previous articles, we explored the meaning of two statistical concepts: that of the P value1 and that of confounding.2 In today’s article, we will focus on another important, though often underrecognized, statistical concept: that of effect modification (sometimes also called “interaction,” although many authors draw a distinction between these terms).3-5 In brief, effect modification is the recognition that the relationship between two variables can be different based on the values of a third variable. Let me illustrate with an example. Suppose that you give a weight-loss drug to a group of people. After a predetermined time, say 3 months, you weigh them to measure the effects of the drug, and you record that on average each individual lost about 10 pounds. Suppose, however, that you are curious to see the effect of the drug in men and women separately. You perform an analysis stratified by sex, and you notice that men lost on average only about 1 pound each, whereas women lost on average about 30 pounds each. What do these results mean? It seems that the effect of the drug on weight loss is different depending on the value of a third variable, namely sex. That is, in this case, sex acts as an effect modifier. Based on this analysis, we may conclude that the drug has real effects on women but not on men.
Before reaching this conclusion, however, it is appropriate to ask whether we have made a mistake. The first obvious thing is to make sure that we have not made a mistake in our collection of data. Once we have excluded the presence of structural bias in our dataset, how can we ascertain that these results, which at eyeballing seem so different, have not been so as a result of chance? Fortunately, we do not have to guess. There is a way to formally test for this hypothesis. If sex is truly an effect modifier, then we can perform what is called in statistical terms an “interaction term” between the exposure (in this case the drug) and the potential effect modifier (in this case sex) in a multivariable model that includes both as exposures. If the P value for that interaction term is less than .05, then the interaction term is statistically significant, and therefore the variable (in this case sex) is confirmed to be an effect modifier. Hence, the results are not due to chance, and the different effects in men and women are plausibly attributable to different biological responses to the medication.
Difference between confounding and effect modification
At this point someone might ask, “What then is the difference between confounding and effect modification? In both cases, we do stratified analysis of the relationship between an exposure (in this case the weight-loss drug) and an outcome (in this case weight loss) based on strata of a third variable (in this case sex).” The difference is fundamental. Confounding, as we explored in a previous article,2 is something that we would like to get rid of. It is the effect of an outside variable on both the exposure and outcome that does not allow us to properly evaluate the real relationship between the two. As such, we try to adjust for this variable, so that its effect is eliminated and we can only observe the relationship between the exposure and the outcome. Effect modification, on the other side, is not something we would like to get rid of. On the contrary, effect modification is part of what we would like to explore and describe because it is part of the biological mechanism that explains the real relationship between the exposure and the outcome. In the example above, if effect modification by sex is confirmed, it implies that there is something in female biology that is not found in male biology (or vice-versa) that makes their response to the medication different and therefore something of interest to study further. Thus, differently from confounding, effect modification is part of the objective reality of the world which we would like to explore and evaluate.
Several questions then arise. How can we know whether a variable is a confounder, an effect modifier, or both? As a general rule of thumb, a confounder would be a variable which, when a stratified analysis is done (or when added to a multivariable model), will change the relationship between the exposure and outcome by 10% or more. However, the relationship between the exposure and the outcome in both strata will be similar. In the example above, it would mean that if sex was only a confounder, then stratifying by sex would show roughly a similar change in the effect (say 9 kg for men and 11 kg for women). An effect modifier on the other hand is one which, when a stratified analysis is done, the association between the exposure and the outcome is very different in the two strata, as illustrated in the example above (for simplicity I am only considering two-level effect modifiers in this article).
Can a variable be both a confounder and an effect modifier? Yes, that is possible. What can be done in this case? The most common approach is to behave the same as when only effect modification is present, namely to show with the interaction term that effect modification exists and present the results between the exposure and outcome separately by the level of the effect modifier (in this example, it means that we need to describe the effects of the weight-loss drug separately for men and women). The stratified analysis/presentation will, by definition, take care of confounding as well.2
Should we always look for effect modification? Not necessarily. As a general rule, we need to test for effect modification only if there is some biological rationale that would compel us to do so, and testing should be hypothesis-driven. A common mistake that some authors make is to perform too many interaction tests and then describe as “positive findings” any test for which P value happens to be less than .05. However, as we pointed out in the article on P value,1 if we perform multiple tests, this increases the probability of false positives, and therefore the probability of spurious findings. Thus, effect modification analysis (with interaction terms) should, generally speaking, be performed with a biological rationale and/or be hypothesis driven.
Conclusion
Effect modification is an essential statistical concept that describes an underlying biological reality in which the association between an exposure and an outcome is different based on the values of a third variable. Differently from confounding, which clouds the association between exposure and outcome, and therefore is something that we try to get rid of, effect modification serves to bring to light a more proper understanding of the biological reality underlying the true association between an exposure and an outcome and as such is something that needs to be explored and described.
Dr. Jovani is assistant professor of medicine, therapeutic endoscopy, digestive diseases, and nutrition at the University of Kentucky Albert B. Chandler Hospital, Lexington. He has no conflicts of interest.
References
1. “The P value: What to make of it? A simple guide for the uninitiated.” GI & Hepatology News. 2019 Sep 23. www.mdedge.com/gihepnews/article/208601/mixed-topics/p-value-what-make-it-simple-guide-uninitiated
2. “Evaluating a paper: Take care not to be confounded.” GI & Hepatology News. 2020 Sep 18. www.mdedge.com/gihepnews/article/228765/mixed-topics/evaluating-paper-take-care-not-be-confounded
3. Corraini P et al. Clin Epidemiol. 2017;9:331-8. doi: 10.2147/CLEP.S162236.
4. VanderWeele TJ. Epidemiology. 2009 Nov;20(6):863-71. doi: 10.1097/EDE.0b013e3181ba333c.
5. Shahar E and Shahar DJ. Epidemiology. 2010 Jul;21(4):587. doi: 10.1097/EDE.0b013e3181e0995c. Author reply 587-8.