Failure to reproduce

In my struggle to keep abreast of all things pediatric, I sample a variety of sources.

Of course each month I scan almost all of the abstracts in the journal Pediatrics. But to get a sense of what the nonmedical community is reading, I begin each morning with a cruise through the electronic versions of the New York Times and the Portland (Maine) Press Herald.

By lunch time I usually have hopscotched my way through the Wall Street Journal. And during our evening adult beverage quiet time, I amuse myself with our local daily. If a news story includes a link to an original article, I usually bore down deep enough to at least read the abstract. Keep in mind that this whole process of keeping current takes little more than a half an hour, 45 minutes tops.

It seems that psychology-related topics dominate the science and medicine stories that I encounter. This shouldn’t surprise you because most of us want to know more about why humans behave the way we do. We also wonder if animal behavior may provide some clues.

It may be because I was trained by careful and skeptical “hard” scientists that I have always read psychosocial and behavioral studies with several grains of salt. Despite my skepticism, I am not beneath embracing the odd study that seems to support one of my biases. The studies that don’t sync with my world view I quickly cast on the rubbish heap because the “sample group was too small,” or the “variables were not adequately controlled for,” or simply because I thought the study was poorly done.

It turns out that my skepticism has not only been well founded, but should have been broader in scope. In a recent study published in the journal Science, three young psychologists undertook a heroic and courageous effort to reproduce 100 studies from three leading psychology journals (Science 2015 Aug 28. doi:10.1126/science.aac4716). Chosen from a larger group, these studies were thought to reflect the core knowledge from which psychologists develop their understanding of such basics as learning, memory, and relationships.

The investigators found that in more than half the studies, they were unable to reproduce the results reported in the original studies despite the fact that in many cases, they were assisted by the original investigators in their attempts to replicate the conditions of the initial studies.

The authors quickly assert that their findings do not suggest that the original investigators were attempting to deceive. Nor does the failure to reproduce results necessarily mean that other future studies might confirm the original findings. Their primary point is that evaluating reproducibility is difficult.

However, this new study is troubling for two reasons. First, it casts even more doubt on the decision to expand the MCAT (Medical College Admission Test) by adding several hours of questions based on psychosocial topics in hopes of creating physicians who are more in tune with the emotional needs and social challenges of their future patients. If the results of more than half of the studies that might be considered the underpinnings of modern psychology can’t be reproduced, are we just asking aspiring medical students to learn a larger collection of half truths? And thus have medical students spend less time learning basic science and developing better critical thinking skills? There are better ways to sort for more empathetic and sensitive physicians than by building an unevenly weighted exam.

Second, although this study highlights the core of what makes science such a powerful and effective tool for discovering the truth, the anti-science folks will point to it as just another example of how we shouldn’t trust anything science tells us.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine, for nearly 40 years. He has authored several books on behavioral pediatrics, including “Coping with a Picky Eater.” Email him at [email protected].

In my struggle to keep abreast of all things pediatric, I sample a variety of sources.

Of course each month I scan almost all of the abstracts in the journal Pediatrics. But to get a sense of what the nonmedical community is reading, I begin each morning with a cruise through the electronic versions of the New York Times and the Portland (Maine) Press Herald.

By lunch time I usually have hopscotched my way through the Wall Street Journal. And during our evening adult beverage quiet time, I amuse myself with our local daily. If a news story includes a link to an original article, I usually bore down deep enough to at least read the abstract. Keep in mind that this whole process of keeping current takes little more than a half an hour, 45 minutes tops.

It seems that psychology-related topics dominate the science and medicine stories that I encounter. This shouldn’t surprise you because most of us want to know more about why humans behave the way we do. We also wonder if animal behavior may provide some clues.

It may be because I was trained by careful and skeptical “hard” scientists that I have always read psychosocial and behavioral studies with several grains of salt. Despite my skepticism, I am not beneath embracing the odd study that seems to support one of my biases. The studies that don’t sync with my world view I quickly cast on the rubbish heap because the “sample group was too small,” or the “variables were not adequately controlled for,” or simply because I thought the study was poorly done.

It turns out that my skepticism has not only been well founded, but should have been broader in scope. In a recent study published in the journal Science, three young psychologists undertook a heroic and courageous effort to reproduce 100 studies from three leading psychology journals (Science 2015 Aug 28. doi:10.1126/science.aac4716). Chosen from a larger group, these studies were thought to reflect the core knowledge from which psychologists develop their understanding of such basics as learning, memory, and relationships.

The investigators found that in more than half the studies, they were unable to reproduce the results reported in the original studies despite the fact that in many cases, they were assisted by the original investigators in their attempts to replicate the conditions of the initial studies.

The authors quickly assert that their findings do not suggest that the original investigators were attempting to deceive. Nor does the failure to reproduce results necessarily mean that other future studies might confirm the original findings. Their primary point is that evaluating reproducibility is difficult.

However, this new study is troubling for two reasons. First, it casts even more doubt on the decision to expand the MCAT (Medical College Admission Test) by adding several hours of questions based on psychosocial topics in hopes of creating physicians who are more in tune with the emotional needs and social challenges of their future patients. If the results of more than half of the studies that might be considered the underpinnings of modern psychology can’t be reproduced, are we just asking aspiring medical students to learn a larger collection of half truths? And thus have medical students spend less time learning basic science and developing better critical thinking skills? There are better ways to sort for more empathetic and sensitive physicians than by building an unevenly weighted exam.

Second, although this study highlights the core of what makes science such a powerful and effective tool for discovering the truth, the anti-science folks will point to it as just another example of how we shouldn’t trust anything science tells us.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine, for nearly 40 years. He has authored several books on behavioral pediatrics, including “Coping with a Picky Eater.” Email him at [email protected].

In my struggle to keep abreast of all things pediatric, I sample a variety of sources.

Of course each month I scan almost all of the abstracts in the journal Pediatrics. But to get a sense of what the nonmedical community is reading, I begin each morning with a cruise through the electronic versions of the New York Times and the Portland (Maine) Press Herald.

By lunch time I usually have hopscotched my way through the Wall Street Journal. And during our evening adult beverage quiet time, I amuse myself with our local daily. If a news story includes a link to an original article, I usually bore down deep enough to at least read the abstract. Keep in mind that this whole process of keeping current takes little more than a half an hour, 45 minutes tops.

It seems that psychology-related topics dominate the science and medicine stories that I encounter. This shouldn’t surprise you because most of us want to know more about why humans behave the way we do. We also wonder if animal behavior may provide some clues.

It may be because I was trained by careful and skeptical “hard” scientists that I have always read psychosocial and behavioral studies with several grains of salt. Despite my skepticism, I am not beneath embracing the odd study that seems to support one of my biases. The studies that don’t sync with my world view I quickly cast on the rubbish heap because the “sample group was too small,” or the “variables were not adequately controlled for,” or simply because I thought the study was poorly done.

It turns out that my skepticism has not only been well founded, but should have been broader in scope. In a recent study published in the journal Science, three young psychologists undertook a heroic and courageous effort to reproduce 100 studies from three leading psychology journals (Science 2015 Aug 28. doi:10.1126/science.aac4716). Chosen from a larger group, these studies were thought to reflect the core knowledge from which psychologists develop their understanding of such basics as learning, memory, and relationships.

The investigators found that in more than half the studies, they were unable to reproduce the results reported in the original studies despite the fact that in many cases, they were assisted by the original investigators in their attempts to replicate the conditions of the initial studies.

The authors quickly assert that their findings do not suggest that the original investigators were attempting to deceive. Nor does the failure to reproduce results necessarily mean that other future studies might confirm the original findings. Their primary point is that evaluating reproducibility is difficult.

However, this new study is troubling for two reasons. First, it casts even more doubt on the decision to expand the MCAT (Medical College Admission Test) by adding several hours of questions based on psychosocial topics in hopes of creating physicians who are more in tune with the emotional needs and social challenges of their future patients. If the results of more than half of the studies that might be considered the underpinnings of modern psychology can’t be reproduced, are we just asking aspiring medical students to learn a larger collection of half truths? And thus have medical students spend less time learning basic science and developing better critical thinking skills? There are better ways to sort for more empathetic and sensitive physicians than by building an unevenly weighted exam.

Second, although this study highlights the core of what makes science such a powerful and effective tool for discovering the truth, the anti-science folks will point to it as just another example of how we shouldn’t trust anything science tells us.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine, for nearly 40 years. He has authored several books on behavioral pediatrics, including “Coping with a Picky Eater.” Email him at [email protected].