Looking at study results with a critical eye

 

As a physician you are the embodiment of delayed gratification. You spent more than 20 years in school before you earned a degree that then allowed you spend another 3-plus years in training before anyone would consider you a “real” doctor. Somewhere along that long and shallow trajectory someone may have said, “You must have done really well on the marshmallow test.”

Described first in 1990 by Shoda et al. in the journal Developmental Psychology, the marshmallow test found that children who could wait longer for a reward (in this case a marshmallow) had higher SAT scores as teenagers. (Dev Psychol. 1990 Nov;26[6]:978-86). In large part because the results of the study feel intuitive and square with our sense of fairness, the marshmallow test has become one of the unquestioned cornerstones of developmental psychology.

That is, until this year, when an attempt to replicate the initial study by Shoda et al. failed to find that the associations between delayed gratification and adolescent achievement were anywhere near as significant as those reported in the 1990 study (Psychol Sci. 2018 May. doi: 10.1177/0956797618761661). Watts et al. suggest that the discrepancy may be explained in part by a failure to adequately control for family background, home environment, and early cognitive ability in the initial experimental design.

Is there a message here? Should we stop wasting our time reading papers from the developmental psychology literature? Not just yet. There are more papers coming out in which the authors attempt to replicate other landmark studies, often without success (“Undergrads Can Improve Psychology,” by Russel T. Warne and Jordan Wagge, The Wall Street Journal, June 20, 2018). Let’s wait and see how much more debunking there is going to be before we throw the baby out with the bath water.

The real message is that every study we encounter should be read with a critical eye regardless of how prestigious the institution of origin and regardless of how much it appeals to our common sense. Our intuition can be a powerful tool when we are looking for answers, but it can lead us astray if we take it too seriously.

It is often said that a good experiment is one that raises more questions than it answers. You don’t have to remember all that stuff you learned when you studied statistics to be able to question the results of a study you read in a peer-reviewed journal. I find that in many of the papers I read I have serious concerns about how well the authors have controlled for the not-so-obvious variables.

So where does this failed attempt at replicating the original marshmallow test study leave us? It is still very likely given your aptitude for delayed gratification that had you been given the test as a preschooler you would not have even touched the marshmallow until the experimenter re-entered the room to end the test and then ... you probably would have offered to share it with her.

 

But these new results suggest that your ability to delay gratification was not some skill with which you were born. You may have been born smarter than the average child, but your skill at delaying gratification may have been one you learned from your parents and assimilated from the home environment in which your grew up. This may be one of those nature-or-nurture skirmishes in which nurture wins. And, it may be another good example of the importance of the first 3 or 4 years in a child’s emotional and psychological development.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine for nearly 40 years. He has authored several books on behavioral pediatrics, including “How to Say No to Your Toddler.” Email him at [email protected].

 

As a physician you are the embodiment of delayed gratification. You spent more than 20 years in school before you earned a degree that then allowed you spend another 3-plus years in training before anyone would consider you a “real” doctor. Somewhere along that long and shallow trajectory someone may have said, “You must have done really well on the marshmallow test.”

Described first in 1990 by Shoda et al. in the journal Developmental Psychology, the marshmallow test found that children who could wait longer for a reward (in this case a marshmallow) had higher SAT scores as teenagers. (Dev Psychol. 1990 Nov;26[6]:978-86). In large part because the results of the study feel intuitive and square with our sense of fairness, the marshmallow test has become one of the unquestioned cornerstones of developmental psychology.

That is, until this year, when an attempt to replicate the initial study by Shoda et al. failed to find that the associations between delayed gratification and adolescent achievement were anywhere near as significant as those reported in the 1990 study (Psychol Sci. 2018 May. doi: 10.1177/0956797618761661). Watts et al. suggest that the discrepancy may be explained in part by a failure to adequately control for family background, home environment, and early cognitive ability in the initial experimental design.

Is there a message here? Should we stop wasting our time reading papers from the developmental psychology literature? Not just yet. There are more papers coming out in which the authors attempt to replicate other landmark studies, often without success (“Undergrads Can Improve Psychology,” by Russel T. Warne and Jordan Wagge, The Wall Street Journal, June 20, 2018). Let’s wait and see how much more debunking there is going to be before we throw the baby out with the bath water.

The real message is that every study we encounter should be read with a critical eye regardless of how prestigious the institution of origin and regardless of how much it appeals to our common sense. Our intuition can be a powerful tool when we are looking for answers, but it can lead us astray if we take it too seriously.

It is often said that a good experiment is one that raises more questions than it answers. You don’t have to remember all that stuff you learned when you studied statistics to be able to question the results of a study you read in a peer-reviewed journal. I find that in many of the papers I read I have serious concerns about how well the authors have controlled for the not-so-obvious variables.

So where does this failed attempt at replicating the original marshmallow test study leave us? It is still very likely given your aptitude for delayed gratification that had you been given the test as a preschooler you would not have even touched the marshmallow until the experimenter re-entered the room to end the test and then ... you probably would have offered to share it with her.

 

But these new results suggest that your ability to delay gratification was not some skill with which you were born. You may have been born smarter than the average child, but your skill at delaying gratification may have been one you learned from your parents and assimilated from the home environment in which your grew up. This may be one of those nature-or-nurture skirmishes in which nurture wins. And, it may be another good example of the importance of the first 3 or 4 years in a child’s emotional and psychological development.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine for nearly 40 years. He has authored several books on behavioral pediatrics, including “How to Say No to Your Toddler.” Email him at [email protected].

 

As a physician you are the embodiment of delayed gratification. You spent more than 20 years in school before you earned a degree that then allowed you spend another 3-plus years in training before anyone would consider you a “real” doctor. Somewhere along that long and shallow trajectory someone may have said, “You must have done really well on the marshmallow test.”

Described first in 1990 by Shoda et al. in the journal Developmental Psychology, the marshmallow test found that children who could wait longer for a reward (in this case a marshmallow) had higher SAT scores as teenagers. (Dev Psychol. 1990 Nov;26[6]:978-86). In large part because the results of the study feel intuitive and square with our sense of fairness, the marshmallow test has become one of the unquestioned cornerstones of developmental psychology.

That is, until this year, when an attempt to replicate the initial study by Shoda et al. failed to find that the associations between delayed gratification and adolescent achievement were anywhere near as significant as those reported in the 1990 study (Psychol Sci. 2018 May. doi: 10.1177/0956797618761661). Watts et al. suggest that the discrepancy may be explained in part by a failure to adequately control for family background, home environment, and early cognitive ability in the initial experimental design.

Is there a message here? Should we stop wasting our time reading papers from the developmental psychology literature? Not just yet. There are more papers coming out in which the authors attempt to replicate other landmark studies, often without success (“Undergrads Can Improve Psychology,” by Russel T. Warne and Jordan Wagge, The Wall Street Journal, June 20, 2018). Let’s wait and see how much more debunking there is going to be before we throw the baby out with the bath water.

The real message is that every study we encounter should be read with a critical eye regardless of how prestigious the institution of origin and regardless of how much it appeals to our common sense. Our intuition can be a powerful tool when we are looking for answers, but it can lead us astray if we take it too seriously.

It is often said that a good experiment is one that raises more questions than it answers. You don’t have to remember all that stuff you learned when you studied statistics to be able to question the results of a study you read in a peer-reviewed journal. I find that in many of the papers I read I have serious concerns about how well the authors have controlled for the not-so-obvious variables.

So where does this failed attempt at replicating the original marshmallow test study leave us? It is still very likely given your aptitude for delayed gratification that had you been given the test as a preschooler you would not have even touched the marshmallow until the experimenter re-entered the room to end the test and then ... you probably would have offered to share it with her.

 

But these new results suggest that your ability to delay gratification was not some skill with which you were born. You may have been born smarter than the average child, but your skill at delaying gratification may have been one you learned from your parents and assimilated from the home environment in which your grew up. This may be one of those nature-or-nurture skirmishes in which nurture wins. And, it may be another good example of the importance of the first 3 or 4 years in a child’s emotional and psychological development.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine for nearly 40 years. He has authored several books on behavioral pediatrics, including “How to Say No to Your Toddler.” Email him at [email protected].