Leif Nelson
Appearances
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It was a tense few months, but in the end, I was allowed to continue doing what I was doing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So I think journals have really complicated incentives.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Of course, they want to publish good work to begin with, so there's some incentive to do some quality check and kind of cover their ass there. But once they've published something, there's a strong incentive for them to defend it or at least to not publicize any errors.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So one of the things the editor-in-chief does is when a manuscript is submitted, I would read it and decide whether it should continue through the peer review process or I could reject it there. And that's called desk rejection.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
One thing I started doing at the journal that wasn't official policy, it was just a practice I decided to adopt, was that when a manuscript was submitted, I would hide the author's names from myself. So I was rejecting things without looking at who the authors were. So the publication committee started a conversation with me, which is totally reasonable, about the overall desk rejection rate.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Am I rejecting too many things, etc.? There was some conversation about whether I was desk rejecting the wrong people. So if I was stepping on important people's toes and an email was forwarded to me from a quote unquote award winning social psychologist, you know, Samin desk rejected my paper. I found this extremely distasteful and I won't be submitting there again.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And when I would try to engage about the substance of my decisions, you know, the scientific basis for them, that wasn't what the conversation was about.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It was a tense few months, but in the end, I was allowed to continue doing what I was doing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We're expanding a team that used to have a different name. We're going to call them the Statistics, Transparency and Rigor editors, the star editors. And so that team will be supplementing the handling editors, the editors who actually organize the peer review and make the decisions on submissions.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Like if a handling editor has a question about the data integrity or about details of the methods or things like that, the star editor team will provide their expertise and help fill in those gaps. We're also, I'm not sure exactly what form this will take, but try to incentivize more accurate and calibrated claims and less hype and exaggeration.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
This is something that I think is particularly challenging with short articles like psychological science publishes and especially, you know, a journal that has really high rejection rate where the vast majority of submissions are rejected. authors are competing for those few spots. And so it feels like they have to make a really bold claim.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And so it's going to be very difficult to play this like back and forth where authors are responding to the perception of what the incentives are. So we need to convey to them that actually, if you go too far, make too bold of claims that aren't warranted, you will be more likely to get rejected.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But I'm not sure if authors will believe that just because we say that they're still competing for a very selective number of spots.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Oh, I don't mind being wrong. I think journalists should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we're only going to publish true things or things that we're 100% sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we'll make mistakes even there.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Sometimes we'll present things with certainty that we shouldn't have presented. What I would like to be involved in and what I plan to do is to encourage more post-publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon, and create more incentives for people to do that and do that well.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Stephen, I'm the person that walks into these academic conferences and everyone is like, here comes Debbie Downer.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Bad question? No. Like, it reminds me of how stressful it all is. We struggle a little bit with thinking about analogies for what we do. We're definitely not police. Police, amongst other things, have institutional power. They have badges, whatever. We don't have any of that. We're not enforcers in any way. The internal affairs thing...
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Hurts a little bit, but I get it, because that's saying, hey, within the behavioral science community, we're the people that are watching the behavioral scientists. And you're right, no one likes internal affairs. Most of our thinking is that we want to be journalists, that it's fun to investigate. That's true for everybody in the field, right?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
They're all curious about whatever it is they're studying. And so we're curious about this. And then when we find things that we think are interesting, we also want to talk about it, not just with each other, but with the outside world.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But I don't identify as much with being a police officer or even a detective, though every now and then people will compare us to something like Sherlock Holmes, and that feels more fun. But in truth, the reason I sort of wince at the question is that the vast majority of the time, it comes with far more burden than it does pleasure.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah. The lawsuit makes all of the psychological burden into a concrete, observable thing. But the part prior to that is that every time we report on anything... That's going to be like, look, we think something bad happened here. Someone is going to be mad at us. And probably more people are going to be. And I don't want people to be mad at me. And I think about some of the people involved.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And it's hard because I know a lot of these people and I know their friends and I know the friends of the friends. And that carries real, real stress for, I think, all three of us.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, that's real bad. I'm not happy with being compared to the Stassi. The optimistic take is that there's less of that than there used to be.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
When any of the three of us go and visit universities, for example, and we talk to doctoral students and we talk to assistant professors and we talk to associate professors, we talk to senior professors, the students basically all behave as though they don't understand why anyone would ever be against what we're saying.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
They wouldn't understand the Stasi thing, but they also wouldn't even understand like why they almost are at the level, I don't understand why we're having you come for a talk. Doesn't everyone already believe this? But when I talk to people that are closer to retirement than they are to being a grad student, they're more like, you know, you're making waves where you don't need to.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
You're pushing back against something that's not there. We've been doing this for decades. Why fix what isn't broken? That sort of thing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
I would say, but it is broken. And your evidence for that would be? The evidence for that is multifold.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Sometimes, for example, we'll get a submission where the research is really solid, but the conclusion is too strong. And I'll sometimes tell authors, hey, look, I'll publish your paper if you tone down the conclusion or even sometimes change the conclusion from saying there is evidence for my hypothesis to there's no evidence one way or the other, but it's still interesting data.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And authors are not always willing to do that, even if it means getting a publication in this journal. So I do think that's a sign that maybe it's a sign that they genuinely believe what they're saying, which is maybe to their credit. I don't know if that's good news or bad news. I think often when we're kind of overselling something, we probably believe what we're saying.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Editors largely in my field are uncompensated for their job, and reviewers are almost purely uncompensated for their job. And so they're all doing it for the love of the field. And those jobs are hard. I'm an occasional reviewer and an occasional editor. And every time I do it, it's basically taxing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
The first part of the job was reading a whole paper and deciding whether the topic was interesting, whether it was contextualized well enough that people would understand what it was about. Whether the study as designed was good at testing the hypothesis as articulated. And only after you get past all of those levels would you say, okay, and now do they have evidence in favor of the hypothesis?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
There were a lot of societal phenomena that we really wanted explanations for, and then social psych offered these kind of easy explanations, or maybe not so easy, but these relatively simple explanations that people wanted to believe just to have an answer and an explanation.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
If you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat.