I’m an educator and journalist who teaches about how to read the news, and I absolutely read to the end! The educator part of me knows that families are such complex organisms that information like income level can’t possibly give you everything you need to know about how well a child will do at X task or Y test. And the journalist part of me knows how little training we all get in actual math and statistics before someone lets us write things about math and statistics. Your work helps to bridge that gap -- and it’s always so useful when it provides a framework for people to do their own critical thinking about a given issue.
I cynically disagree that the authors of these papers think adjusting for demographics fixes the problem. They need to “publish or perish” so they publish and don’t worry about it too much. Then they don’t teach their grad students to carefully analyze their results. You don’t get papers published by calling your research not significant right at the top. Academia is absolutely broken.
Hi Emily. I’m not an economist and I read to the end! I may not have a solid grasp of everything, but you explain it so well that it still makes sense to those without a research background. I appreciate you!
My issue with the “it’s not causal, so we can disregard” takeaway comes from Annie Duke’s book Thinking In Bets. You don’t know the true effect but you can still say the effect is negative-to-neutral. So you can take action (avoid the processed foods) and the result will be that either A/ it’s a waste of effort, or B/ it’s worth it (prevents cancer). That is a better risk profile than taking no action and the result is either A/ waste of effort, or B/ big mistake (causes cancer).
As someone who teaches college statistics, I love your deep dives. You explain things at such an understandable level; and, it helps me bring in these types of discussions to my students. Thanks!
Dear Emily, thank you for your wonderful newsletter, I have been lucky to have my babies after you wrote your 3 books and they have been so helpful during my pregnancies and the early years. I am looking forward to your book about teenagers!
Regarding breastfeeding, this has been a tricky issue in my household. Breastfeeding was a nightmare and impossible to establish with one child, but it was very easy and smooth with the other child. Thanks to your analysis, this difference has not something I have been worrying about very much with regards to future outcomes for my kids. However, I came across a recent paper by Fitzsimmons and Vera-Hernandez published in AEJ: applied in July 22 and the story is a little different there. Using quasi-experimental evidence the authors do appear to find a significant impact on cognitive skills in the first 7 years of life. The population of study are low-income families only, so the external validity is still unclear. Being an applied economist myself, I am pretty convinced by their causal setup, so I am back to worrying about not having breastfed one of my kids. What do you think about this study in the context of the wider literature on breastfeeding? Any discussion would be welcome, thank you!!!
I believe that these studies exist because there is a market for them. People want to feel in control of their lives. Lifestyle magazines need content to write about. Sociopaths need justifications for judging others.
Furthermore, the subject matter of these studies is enmeshed with cultural values and issues of personal identity. A contrary result might be deeply threatening to an individual or group. In some respects, it is the utter lack of an effect that makes this subject matter so useful for the role of cultural signifier. Choices that matter--choices that have immediate, obvious consequences--are not suited for this cultural signifier role, because everyone will necessarily end up making the same choice.
In the modern era, authority derives from The Science as opposed to The Bible. These studies exist to provide the required footnote on an argument that the proponent wanted to make anyway, and would have made anyway.
Great explanation! I would love if you had Andrew Huberman on your podcast (or vice versa) so we could hear you discuss in real time. I think it would be one of the most interesting conversations possibly ever about data!
This was a great read, but I find myself frustrated at the lack of black and white when it comes to data analysis and interpretation. What is the actionable step here--it seems that you can argue everything down to potentially confounding somehow. So, the answer always is "it depends" and you have to do risk management assessment and personal weighing of tradeoffs. I always wish there was an easy button and way to definitively say "YES or NO.
Hey Emily, I love your stuff and agree most of the time. I struggle sometimes with then what we do in public health when all we have is observational data. In many cases, it can provide actionable information but it is always challenging to ensure all messengers and users are good stewards of the information. I know the answer isn't "we should stop using all observational data", but I'm wondering where you find this type of information useful. I've always felt that starting with less cost and time intensive observational data can make the case for RCTs or other more statistically rigorous studies. Just curious where you see this information HELPFUL rather than HARMFUL.
Love this and it is exactly what undergraduates students need to understand about statistics. I will use it in my research methods course at my university this semester! Thank you, Emily!
Thanks for the great read, Emily! I have two follow-up questions:
1. Is there a good summary of the "steelman" version of the opposing argument that you would recommend (i.e., that we can reliably approximate an RCT through controlling observed differences between non-randomized groups)? I'm inclined to agree with your argument, but I'd be interested in reading the best possible version of the opposite viewpoint.
2. Do you have any recommended books on this topic? I took every statistics/data course that I could get my hands on during my MBA program a few years back, but I'm always looking for opportunities to learn more.
Great read! I feel like my ideal econometrics paper would be something that generalizes your point, perhaps something that looked at the results of a large number of observational studies that were later examined using randomized or better-controlled trials.
This is probably the best article I've read by you. I appreciate you diving into the details on RCTs versus observational studies. As someone who works in medicine (psychiatry), I have problems with RCTs, even though they are "the best studies" we can run other than meta-analyses. This article applies to the many health opinions that swing back and forth, like if eggs are healthy, if I should be prescribing vitamin D for depression, etc.
I would love to hear a respectful debate, maybe in podcast form, with you and Andrew Huberman on this topic. You both seem grounded and humble enough to have a fruitful discussion.
I’m an educator and journalist who teaches about how to read the news, and I absolutely read to the end! The educator part of me knows that families are such complex organisms that information like income level can’t possibly give you everything you need to know about how well a child will do at X task or Y test. And the journalist part of me knows how little training we all get in actual math and statistics before someone lets us write things about math and statistics. Your work helps to bridge that gap -- and it’s always so useful when it provides a framework for people to do their own critical thinking about a given issue.
I cynically disagree that the authors of these papers think adjusting for demographics fixes the problem. They need to “publish or perish” so they publish and don’t worry about it too much. Then they don’t teach their grad students to carefully analyze their results. You don’t get papers published by calling your research not significant right at the top. Academia is absolutely broken.
Hi Emily. I’m not an economist and I read to the end! I may not have a solid grasp of everything, but you explain it so well that it still makes sense to those without a research background. I appreciate you!
My issue with the “it’s not causal, so we can disregard” takeaway comes from Annie Duke’s book Thinking In Bets. You don’t know the true effect but you can still say the effect is negative-to-neutral. So you can take action (avoid the processed foods) and the result will be that either A/ it’s a waste of effort, or B/ it’s worth it (prevents cancer). That is a better risk profile than taking no action and the result is either A/ waste of effort, or B/ big mistake (causes cancer).
As someone who teaches college statistics, I love your deep dives. You explain things at such an understandable level; and, it helps me bring in these types of discussions to my students. Thanks!
Dear Emily, thank you for your wonderful newsletter, I have been lucky to have my babies after you wrote your 3 books and they have been so helpful during my pregnancies and the early years. I am looking forward to your book about teenagers!
Regarding breastfeeding, this has been a tricky issue in my household. Breastfeeding was a nightmare and impossible to establish with one child, but it was very easy and smooth with the other child. Thanks to your analysis, this difference has not something I have been worrying about very much with regards to future outcomes for my kids. However, I came across a recent paper by Fitzsimmons and Vera-Hernandez published in AEJ: applied in July 22 and the story is a little different there. Using quasi-experimental evidence the authors do appear to find a significant impact on cognitive skills in the first 7 years of life. The population of study are low-income families only, so the external validity is still unclear. Being an applied economist myself, I am pretty convinced by their causal setup, so I am back to worrying about not having breastfed one of my kids. What do you think about this study in the context of the wider literature on breastfeeding? Any discussion would be welcome, thank you!!!
Thank you, Emily. Keep fighting the good fight.
I believe that these studies exist because there is a market for them. People want to feel in control of their lives. Lifestyle magazines need content to write about. Sociopaths need justifications for judging others.
Furthermore, the subject matter of these studies is enmeshed with cultural values and issues of personal identity. A contrary result might be deeply threatening to an individual or group. In some respects, it is the utter lack of an effect that makes this subject matter so useful for the role of cultural signifier. Choices that matter--choices that have immediate, obvious consequences--are not suited for this cultural signifier role, because everyone will necessarily end up making the same choice.
In the modern era, authority derives from The Science as opposed to The Bible. These studies exist to provide the required footnote on an argument that the proponent wanted to make anyway, and would have made anyway.
Great explanation! I would love if you had Andrew Huberman on your podcast (or vice versa) so we could hear you discuss in real time. I think it would be one of the most interesting conversations possibly ever about data!
This was a great read, but I find myself frustrated at the lack of black and white when it comes to data analysis and interpretation. What is the actionable step here--it seems that you can argue everything down to potentially confounding somehow. So, the answer always is "it depends" and you have to do risk management assessment and personal weighing of tradeoffs. I always wish there was an easy button and way to definitively say "YES or NO.
Thanks for the wonderful work on this. Residual confounding should be one of those mental building blocks everyone in a modern society knows.
Hey Emily, I love your stuff and agree most of the time. I struggle sometimes with then what we do in public health when all we have is observational data. In many cases, it can provide actionable information but it is always challenging to ensure all messengers and users are good stewards of the information. I know the answer isn't "we should stop using all observational data", but I'm wondering where you find this type of information useful. I've always felt that starting with less cost and time intensive observational data can make the case for RCTs or other more statistically rigorous studies. Just curious where you see this information HELPFUL rather than HARMFUL.
Great break down of some very complicated topics!
Love this and it is exactly what undergraduates students need to understand about statistics. I will use it in my research methods course at my university this semester! Thank you, Emily!
Thanks for the great read, Emily! I have two follow-up questions:
1. Is there a good summary of the "steelman" version of the opposing argument that you would recommend (i.e., that we can reliably approximate an RCT through controlling observed differences between non-randomized groups)? I'm inclined to agree with your argument, but I'd be interested in reading the best possible version of the opposite viewpoint.
2. Do you have any recommended books on this topic? I took every statistics/data course that I could get my hands on during my MBA program a few years back, but I'm always looking for opportunities to learn more.
Great read! I feel like my ideal econometrics paper would be something that generalizes your point, perhaps something that looked at the results of a large number of observational studies that were later examined using randomized or better-controlled trials.
This is probably the best article I've read by you. I appreciate you diving into the details on RCTs versus observational studies. As someone who works in medicine (psychiatry), I have problems with RCTs, even though they are "the best studies" we can run other than meta-analyses. This article applies to the many health opinions that swing back and forth, like if eggs are healthy, if I should be prescribing vitamin D for depression, etc.
I would love to hear a respectful debate, maybe in podcast form, with you and Andrew Huberman on this topic. You both seem grounded and humble enough to have a fruitful discussion.