Not the point of this post at all, but I just realized imma is a really efficient contraction. That’s four words crunched down into one. I’d hate to have to learn English as a second language.
50% of all published papers in Psychology are not reproducible …
ssssh
The replication crisis is real, but I’m going to give some pushback on the “ssssh” like it’s some kind of conspiracy “they” don’t want you to know about™. We live in an era of unprecedented and extremely dangerous anti-intellectualism, and pushing this as some kind of conspiracy is honestly really gross.
- The entire reason the crisis became known is because scientists have and are having the integrity to try to replicate results from existing studies. They want the science in their field to be sound, and they’ve been extremely vocal about this problem from the minute they found it. This wasn’t some “whistleblower” situation.
- Arguably a major reason why it took so long for this to come to the fore is because government agencies which administer grants focus much less on replicating previous experiments and more on “new” stuff. This would ironically be much less of a problem if more funds were allocated for scientific research (i.e. so they weren’t so competitive that researchers feel the need to publish “new” research lest their request be denied). This “ssssh” rhetoric makes the voting public want the exact opposite of that because it tells them that their tax dollars are being funneled into some conspiratorial financial black hole.
- This happens in large part because concrete, replicable research on humans is extremely hard, not because the researchers lack integrity and just want to publish slop. In CS, I can control for basically everything on my computer and give you a mathematical proof that what I wrote works for everything every time. In physics, I can give exact parameters for my simulation or literal schematics for my device. A psychological or sociological experiment is vastly more difficult to remove confounding variables from or to properly document the confounding variables in.
- This doesn’t invalidate soft sciences like anti-intellectuals would want you to believe. While some specific studies may not be replicable, this is why meta-analyses and systematic reviews are so important in medicine, psychology, sociology, etc.: they give the “average” of the existing literature on a specific subject, so outliers get discovered, and there’s far more likelihood that their results are correct or close to correct.
- This is actively being worked on, and researchers are more aware of it than ever – making them more cognizant of the way they design their experiments and discuss their methodologies.
- One of the major reasons for problems with replication isn’t actually that the original studies were bunk within the population they were sampling. Rather, it’s that once replication was attempted on people from diverse cultures rather than the narrow range of cultures often sampled in many (especially older) papers (“Western, educated, industrialized, rich, democratic”), the significance observed disappeared. As noted in the linked article, 50% given that fact is actually not half-bad. With much more extensive globalization in the modern day and a larger awareness of this problem, it should become less and less severe.
EDIT: I just noticed that they also got their facts wrong in a subtle but meaningful way: the statistic is that 50% of the published papers aren’t replicable, not reproducible. Reproducibility is taking an existing dataset and using it to reach the same conclusions. For example, if I have a dataset of 500 pictures of tires and publish “Tires: Are they mostly round and black?” in Tireology, claiming based on the dataset that tires are usually round and black, then I would hope that Scientist B. couldn’t take that same dataset of 500 tire pictures and come to the conclusion that they’re usually square and blue. However, replication would be if Scientist B. got their own new dataset of say 800 tire pictures and attempted to reach my same findings. If they found from this dataset that tires are usually square and blue but found from my dataset that they’re usually round and black, then my results would be reproducible but not replicable. If Scientist B. got the same results as me from the new dataset, then my results would be replicable, but it wouldn’t say anything about reproducibility. Here, a lack of replication might come from taking too narrow a sample of tires (I found the tires by camping out in a McDonald’s parking lot in Norfolk, Nebraska over the course of a weekend), that I published my findings in 1985 but that 40 years later tires really have changed, that there was some issue with how I took the pictures, etc.
I don’t consider Psychology to be a scientific discipline - I belong to the hard sciences crowd.
My wife is a psychologist.
You understand that the “hard sciences” are also affected by this crisis, correct? “Soft science” is a borderline meaningless term that stigmatizes entire fields of science to the sole benefit of anti-intellectuals.
Even when we take into consideration that the problem is currently worse in sciences like psychology, economics, sociology, etc.: “these results support the scientific status of the social sciences against claims that they are completely subjective, by showing that, when they adopt a scientific approach to discovery, they differ from the natural sciences only by a matter of degree.” Social sciences are science.
You don’t belong to “the hard sciences crowd”; you belong to a Sheldon Cooper-esque stereotype who devalues work you don’t understand.
Love the write-up, well done. These issues are huge, complex, fascinating, and depressing. It’s always worth defending science, and you’re right - this is basically the opposite of a conspiracy. Experts are actively screaming “something is wrong here!”
But, yeah, wow. What a shit take. Psychology is not science, from someone married to a psychologist? Soft sciences aren’t science?
Correct me if I’m wrong, but Charles Darwin’s science was as soft as it gets! He didn’t have p-values, he had pretty birds with funky beaks. One of the most important scientists to ever live, and his masterpiece did not have a single quantitative model.
Just because psychology got irrationally stuck on Freud for so long doesn’t mean it’s not science. We all learn about Lamarckian inheritance and think it’s goofy as shit, doesn’t mean we dismiss the entire field of biology.
No, the difference in the replication crisis between the soft “sciences” and the hard is enormous. The soft are basically producing results equal to making coin tosses.
You have clearly never actually done “hard sciences” research in any meaningful way if this is your take. And computer science does not count as a science at all, it is more like engineering. Mathematics is a “hard science” that can be implemented through computer science, and physics is a “hard science” that can be implemented through electrical engineering (and as a subset computer engineering).
But even then mathematics is closer to philosophy and logic than any of the physical sciences. The physical sciences like physics, chem, bio are very different due to their experimental nature, and how sensitive they can be to specific conditions of the experiments. And the more complex the system being studied is, the harder it is to control variability which is why the social sciences like psychology and economics are working on incredibility difficult problems in systems we do not currently fully understand, and are more vulnerable to difficult reproducing and replicating the conclusions.
This is in contrast to computer science where we fully understand the system because humans have built it, and it is a machine built on the principles discovered by physicists and implemented by electrical engineers to run calculations that are created by mathematicians.
I’m sure you know very well that what I wrote is perfectly true, so why the essay pretending otherwise?
I wonder if she regrets her marriage, or if she’s trying to fix you.
Hey, maybe it’s both!
I did my masters thesis on high fat diets and while I was doing my lit review I realized there was no standard for what a “high fat diet” even is. There are SO many variables and its insane some of the logic leaps some studies come to to complete a narrative.
Out of curiosity, how did you decide to define it and why?
We used purified high fat diets, one at 40% and one at 60% and compared the two. We had a whole other project where each group were supplemented with lentils but we I focussed on just the difference between those two diets where the only variable between them were the carb/fat percentage, they were otherwise the same/pure.
That’s interesting. Was compliance difficult? I work with dieticians, and they have all mentioned difficulty with compliance. Americans and food. 🤷♂️
Compliance wasn’t an issue since we we ran the study in mice and they all liked the food. they’re all basically clones so so it helps eliminate a LOT of variables. As expected we found the 60% diet induced a much more dramatic phenotype than the 40% but both induced obesity in general, but even ONLY having 60 vs 40% fat the differences were significant enough to make me reluctant to compare the two HFDs especially when you dive into microbiota stuff. I wouldn’t say its apples and oranges, more like apples and crab apples… or something.
Oh, ok. I just assumed a human study, but mice makes more sense. It certainly sounds like an interesting study. I find nutrition to be an engaging topic, especially considering the availability of choices that many have now. Thank you for answering my questions!
no worries, I love talking about work! Nutrition is especially interesting given how relevant it is in our day to day lives and how complicated everything is between food itself, genetics and our gut microbiome. I could read about it all day, and not because I had to for two years!
So do you recommend a high fat diet?
60% produced a more dramatic phenotype and I remember it being the most popular diet in animal studies (I did all this 10 years ago so the details are a little fuzzy) so I’d probably go with that one.
Can you explain what you mean by a more dramatic phenotype?
In animal research we often refer to genotype and phenotype. Genotype refers to the set of genes the animals the animals carry (what they are capable of expressing) and phenotype refers to the physical/clinical expression/presentation/characteristics of the animal or disease state. My guys were all “wild type” meaning they’re just “normal” standard mice and we induced the “obese phenotype” (obese disease state with the associated characteristics and physical presentation associated with the disease) with the two high fat diets. 60% had a greater impact on inducing these changes compared to the control group than the 40% group.
**Replication crisis intensifies
P-hacking?
Interpreters gotta interpret.
“I mean it’s all just numbers, right?”
Psychology has an embarrassing history.
Half of their studies aren’t reproducible. Their most famous study is basically a fraud. They’re behind lobotomies, the satanic panic, and Eugenics.
I’m not anti-intelectual or a Scientologist or anything. I guess what I’m trying to say is that psychologists like Jordan Peterson might want to clean up their own room before trying to lecture the rest of us.
Psychology has an embarrassing history.
It really doesn’t?
Half their studies aren’t reproducible.
Replicable*, and also see here.
Their most famous study is basically a fraud.
Do you mean the Stanford prison experiment, which is famous because of how terrible it was? The one that’s taught in Psych 101 classes as a lesson on ethics and how not to design an experiment? Because while I would argue it’s not the most famous study, the entire reason it’s famous is because it was so shittily designed that psychologists going forward took lessons from it. No one’s holding that up to say “Wow, look at this great study we, the field of psychology, collectively did.”
They’re behind lobotomies
That was psychiatry and neurology, but I don’t expect you to know the difference.
They’re behind the Satanic panic
That a random quack psychiatrist came out and publicized this doesn’t mean that “the field of psychology” is behind the Satanic panic. Dr. Oz is a fraud who used his platform to sell bullshit supplements; does that make the field of medicine “behind” homeopathy?
They’re behind eugenics
This literally isn’t true, or at least it’s a ridiculous half-truth to put psychology at the forefront of eugenics. Eugenics is – surprise, surprise – rooted in biology after inheritance became more widely understood (read: we knew just enough to be dangerous). Eugenics had its hand in basically every natural science, and so you’ll find occasional psychologists like Henry H. Goddard showing up, but you’ll see biologists, statisticians, politicians, and so forth. Eventually eugenics spread into fields like psychiatry (note: different from psychology), but “they’re behind eugenics” is absolute fucking horseshit that you fail to back up with literally anything.
I’m not anti-intelectual [sic] or a Scientologist or anything
Uh-huh…
I guess what I’m trying to say is that psychologists like Jordan Peterson might want to clean up their own room before trying to lecture the rest of us.
Why are you bringing up Jordan Peterson? Peterson is widely despised among psychologists, he no longer works at the University of Toronto, and instead of contributing research to the field or engaging in clinical practice, he puts out self-help sludge. “I’m not an anti-intelectual, but I’m going to take an entire century-old field of science and compress it into Philip Zimbardo(?) and Jordan Peterson so I can say that science bad actually.”
