Leif Nelson
Appearances
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
My name is Leif Nelson, and I'm a professor of business administration at University of California, Berkeley.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
We started our blog in late 2013. We decided we wanted to have a blog because we thought it would be fun to write things that were shorter than a journal article and that we did not have to wait two and a half years for the review process to play out. And so with that in mind, we just needed to name it and we wanted something that would be related to what we do. That's maybe the data part.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
but would definitely not be sending signals of self-seriousness. So we tried out a few things, and somewhere in there, De De Colato was one that we obviously landed on. It had this nice entertaining feature that Yuri is Chilean. And so when he had suggested the name... He thought it rhymed, which still tickles me and Joe, because for him it's data colada.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
The classic forms of what we'd characterized as p-hacking, they're not quite errors, they're decisions that are accidentally self-serving. It's if you measure multiple things but only report the one you like the most.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Or you run a study where there's three treatments, condition A, condition B, and condition C, but in the end you drop condition B and you don't even talk about it, you just compare A to C.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And then there are things that are mildly statistical, but in a very relaxed way. Well, we collected this data, but it's kind of skewed. It has some outliers. and you say, we should eliminate those outliers, or we should Windsorize the outliers, which is basically truncating them down to a lower high number.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Or you could run them through an algorithm where you say, oh, let's transform them with a logarithm or with a square root. And those are all decisions that are justifiable. They're not crazy. It's just, if you have a consideration of reporting one variable or the other,
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
and one variable makes your hypothesis look good, and the other variable makes your hypothesis look less good, you end up reporting the one that looks good, either because you're being self-serving, or honestly, because you'd say, like, I'm not sure which one is better, but my hypothesis tells me it should be the one that looks good, and that one looks good. It's probably the better measure.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
These will be things that can be as simple as a typo. where someone's writing up their report and the means are actually 5.1 and 5.12, but instead someone writes it down as 51.2. And you're like, wow, that's a huge effect, right? And no one corrects it because it's a huge effect in the direction that they were expecting. And so literally a typo might end up in print.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And that's before we get to anything like fraud, like the active fabrication of data or manipulation of data.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Yeah. Well, Stephen, you were asking a question that is pretty heavy and one that I'm not particularly well-equipped to answer. If you'd asked me five years ago, I think I would have been more refined in my answer and I would have said, no, that's not a slippery slope problem. There's a slippery slope between I collect five measures and report one versus I collect 10 measures and I report one.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
That's slippery slope. But making up data feels qualitatively different. And I still largely stand by that view. But there have been enough anecdotes that other people, whistleblower types, have presented to us that sound a lot more like someone says, yeah, you know, at first you do the thing where you drop some measures or drop a condition or you remove the outliers.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And then also you take participant 35 and you change their answer from a 7 to a 9. You're like, whoa, that last one doesn't sound the same. But maybe there's some psychology for that, that it feels like it's an extension.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
The very first blog post we posted was about identification of fraudulent data in a paper published 10 years ago. And that one was discovered because Yuri had made a chart for a totally different paper where he was mining data from multiple published studies to just make a chart. And I looked at his figure of this other research group's data and said, that seems unusual.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
I want to go read that paper. And so I read the paper and then looked at that data set. In that one, it had collected data on a nine-point interval scale, so people can answer one, two, three, up through nine. And there were numbers in the data set that were things like negative 1.7. And so you say, oh, okay, we're done. Nothing fancy.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Once you open the data set, you can then close it and say it's broken.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Bridging the gap between those two positions is this other entity, Harvard University. We only know what they've said outwardly, which is that they've put her on administrative leave, and they've recommended the retraction of those four papers, or the retraction of three plus an amendment to a previously retracted paper.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Certainly scary. Scary because it's so unfamiliar. I found out from talking, basically I was exchanging emails with a reporter. And so between these emails, she came back to me and was like, well, now given the lawsuit, would you like to add a new comment? And I was basically like, what, what lawsuit are you talking about?
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And so it's like this devastating thing to be like, oh my God, it's like the whole house is collapsing and no one told me.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
She was at the center of everything. Being a prestigious faculty member at Harvard and all of her public speaking and her books. Her reputation was perfect. She was synonymous with the highest levels of research in organizational behavior. She's just a giant in the field.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It was a tense few months, but in the end, I was allowed to continue doing what I was doing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So I think journals have really complicated incentives.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Of course, they want to publish good work to begin with, so there's some incentive to do some quality check and kind of cover their ass there. But once they've published something, there's a strong incentive for them to defend it or at least to not publicize any errors.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So one of the things the editor-in-chief does is when a manuscript is submitted, I would read it and decide whether it should continue through the peer review process or I could reject it there. And that's called desk rejection.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
One thing I started doing at the journal that wasn't official policy, it was just a practice I decided to adopt, was that when a manuscript was submitted, I would hide the author's names from myself. So I was rejecting things without looking at who the authors were. So the publication committee started a conversation with me, which is totally reasonable, about the overall desk rejection rate.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Am I rejecting too many things, etc.? There was some conversation about whether I was desk rejecting the wrong people. So if I was stepping on important people's toes and an email was forwarded to me from a quote unquote award winning social psychologist, you know, Samin desk rejected my paper. I found this extremely distasteful and I won't be submitting there again.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And when I would try to engage about the substance of my decisions, you know, the scientific basis for them, that wasn't what the conversation was about.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It was a tense few months, but in the end, I was allowed to continue doing what I was doing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We're expanding a team that used to have a different name. We're going to call them the Statistics, Transparency and Rigor editors, the star editors. And so that team will be supplementing the handling editors, the editors who actually organize the peer review and make the decisions on submissions.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Like if a handling editor has a question about the data integrity or about details of the methods or things like that, the star editor team will provide their expertise and help fill in those gaps. We're also, I'm not sure exactly what form this will take, but try to incentivize more accurate and calibrated claims and less hype and exaggeration.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
This is something that I think is particularly challenging with short articles like psychological science publishes and especially, you know, a journal that has really high rejection rate where the vast majority of submissions are rejected. authors are competing for those few spots. And so it feels like they have to make a really bold claim.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And so it's going to be very difficult to play this like back and forth where authors are responding to the perception of what the incentives are. So we need to convey to them that actually, if you go too far, make too bold of claims that aren't warranted, you will be more likely to get rejected.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But I'm not sure if authors will believe that just because we say that they're still competing for a very selective number of spots.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Oh, I don't mind being wrong. I think journalists should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we're only going to publish true things or things that we're 100% sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we'll make mistakes even there.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Sometimes we'll present things with certainty that we shouldn't have presented. What I would like to be involved in and what I plan to do is to encourage more post-publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon, and create more incentives for people to do that and do that well.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Stephen, I'm the person that walks into these academic conferences and everyone is like, here comes Debbie Downer.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Bad question? No. Like, it reminds me of how stressful it all is. We struggle a little bit with thinking about analogies for what we do. We're definitely not police. Police, amongst other things, have institutional power. They have badges, whatever. We don't have any of that. We're not enforcers in any way. The internal affairs thing...
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Hurts a little bit, but I get it, because that's saying, hey, within the behavioral science community, we're the people that are watching the behavioral scientists. And you're right, no one likes internal affairs. Most of our thinking is that we want to be journalists, that it's fun to investigate. That's true for everybody in the field, right?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
They're all curious about whatever it is they're studying. And so we're curious about this. And then when we find things that we think are interesting, we also want to talk about it, not just with each other, but with the outside world.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But I don't identify as much with being a police officer or even a detective, though every now and then people will compare us to something like Sherlock Holmes, and that feels more fun. But in truth, the reason I sort of wince at the question is that the vast majority of the time, it comes with far more burden than it does pleasure.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah. The lawsuit makes all of the psychological burden into a concrete, observable thing. But the part prior to that is that every time we report on anything... That's going to be like, look, we think something bad happened here. Someone is going to be mad at us. And probably more people are going to be. And I don't want people to be mad at me. And I think about some of the people involved.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And it's hard because I know a lot of these people and I know their friends and I know the friends of the friends. And that carries real, real stress for, I think, all three of us.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, that's real bad. I'm not happy with being compared to the Stassi. The optimistic take is that there's less of that than there used to be.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
When any of the three of us go and visit universities, for example, and we talk to doctoral students and we talk to assistant professors and we talk to associate professors, we talk to senior professors, the students basically all behave as though they don't understand why anyone would ever be against what we're saying.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
They wouldn't understand the Stasi thing, but they also wouldn't even understand like why they almost are at the level, I don't understand why we're having you come for a talk. Doesn't everyone already believe this? But when I talk to people that are closer to retirement than they are to being a grad student, they're more like, you know, you're making waves where you don't need to.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
You're pushing back against something that's not there. We've been doing this for decades. Why fix what isn't broken? That sort of thing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
I would say, but it is broken. And your evidence for that would be? The evidence for that is multifold.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Sometimes, for example, we'll get a submission where the research is really solid, but the conclusion is too strong. And I'll sometimes tell authors, hey, look, I'll publish your paper if you tone down the conclusion or even sometimes change the conclusion from saying there is evidence for my hypothesis to there's no evidence one way or the other, but it's still interesting data.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And authors are not always willing to do that, even if it means getting a publication in this journal. So I do think that's a sign that maybe it's a sign that they genuinely believe what they're saying, which is maybe to their credit. I don't know if that's good news or bad news. I think often when we're kind of overselling something, we probably believe what we're saying.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Editors largely in my field are uncompensated for their job, and reviewers are almost purely uncompensated for their job. And so they're all doing it for the love of the field. And those jobs are hard. I'm an occasional reviewer and an occasional editor. And every time I do it, it's basically taxing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
The first part of the job was reading a whole paper and deciding whether the topic was interesting, whether it was contextualized well enough that people would understand what it was about. Whether the study as designed was good at testing the hypothesis as articulated. And only after you get past all of those levels would you say, okay, and now do they have evidence in favor of the hypothesis?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
There were a lot of societal phenomena that we really wanted explanations for, and then social psych offered these kind of easy explanations, or maybe not so easy, but these relatively simple explanations that people wanted to believe just to have an answer and an explanation.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
If you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat.