Leif Nelson

My name is Leif Nelson, and I'm a professor of business administration at University of California, Berkeley.

My name is Leif Nelson, and I'm a professor of business administration at University of California, Berkeley.

We started our blog in late 2013. We decided we wanted to have a blog because we thought it would be fun to write things that were shorter than a journal article and that we did not have to wait two and a half years for the review process to play out. And so with that in mind, we just needed to name it and we wanted something that would be related to what we do. That's maybe the data part.

We started our blog in late 2013. We decided we wanted to have a blog because we thought it would be fun to write things that were shorter than a journal article and that we did not have to wait two and a half years for the review process to play out. And so with that in mind, we just needed to name it and we wanted something that would be related to what we do. That's maybe the data part.

but would definitely not be sending signals of self-seriousness. So we tried out a few things, and somewhere in there, De De Colato was one that we obviously landed on. It had this nice entertaining feature that Yuri is Chilean. And so when he had suggested the name... He thought it rhymed, which still tickles me and Joe, because for him it's data colada.

but would definitely not be sending signals of self-seriousness. So we tried out a few things, and somewhere in there, De De Colato was one that we obviously landed on. It had this nice entertaining feature that Yuri is Chilean. And so when he had suggested the name... He thought it rhymed, which still tickles me and Joe, because for him it's data colada.

The classic forms of what we'd characterized as p-hacking, they're not quite errors, they're decisions that are accidentally self-serving. It's if you measure multiple things but only report the one you like the most.

The classic forms of what we'd characterized as p-hacking, they're not quite errors, they're decisions that are accidentally self-serving. It's if you measure multiple things but only report the one you like the most.

Or you run a study where there's three treatments, condition A, condition B, and condition C, but in the end you drop condition B and you don't even talk about it, you just compare A to C.

Or you run a study where there's three treatments, condition A, condition B, and condition C, but in the end you drop condition B and you don't even talk about it, you just compare A to C.

And then there are things that are mildly statistical, but in a very relaxed way. Well, we collected this data, but it's kind of skewed. It has some outliers. and you say, we should eliminate those outliers, or we should Windsorize the outliers, which is basically truncating them down to a lower high number.

And then there are things that are mildly statistical, but in a very relaxed way. Well, we collected this data, but it's kind of skewed. It has some outliers. and you say, we should eliminate those outliers, or we should Windsorize the outliers, which is basically truncating them down to a lower high number.

Or you could run them through an algorithm where you say, oh, let's transform them with a logarithm or with a square root. And those are all decisions that are justifiable. They're not crazy. It's just, if you have a consideration of reporting one variable or the other,

Or you could run them through an algorithm where you say, oh, let's transform them with a logarithm or with a square root. And those are all decisions that are justifiable. They're not crazy. It's just, if you have a consideration of reporting one variable or the other,

and one variable makes your hypothesis look good, and the other variable makes your hypothesis look less good, you end up reporting the one that looks good, either because you're being self-serving, or honestly, because you'd say, like, I'm not sure which one is better, but my hypothesis tells me it should be the one that looks good, and that one looks good. It's probably the better measure.

and one variable makes your hypothesis look good, and the other variable makes your hypothesis look less good, you end up reporting the one that looks good, either because you're being self-serving, or honestly, because you'd say, like, I'm not sure which one is better, but my hypothesis tells me it should be the one that looks good, and that one looks good. It's probably the better measure.

And here's Simonson.

And here's Simonson.

These will be things that can be as simple as a typo. where someone's writing up their report and the means are actually 5.1 and 5.12, but instead someone writes it down as 51.2. And you're like, wow, that's a huge effect, right? And no one corrects it because it's a huge effect in the direction that they were expecting. And so literally a typo might end up in print.

These will be things that can be as simple as a typo. where someone's writing up their report and the means are actually 5.1 and 5.12, but instead someone writes it down as 51.2. And you're like, wow, that's a huge effect, right? And no one corrects it because it's a huge effect in the direction that they were expecting. And so literally a typo might end up in print.

And that's before we get to anything like fraud, like the active fabrication of data or manipulation of data.

And that's before we get to anything like fraud, like the active fabrication of data or manipulation of data.

Yeah. Well, Stephen, you were asking a question that is pretty heavy and one that I'm not particularly well-equipped to answer. If you'd asked me five years ago, I think I would have been more refined in my answer and I would have said, no, that's not a slippery slope problem. There's a slippery slope between I collect five measures and report one versus I collect 10 measures and I report one.

Yeah. Well, Stephen, you were asking a question that is pretty heavy and one that I'm not particularly well-equipped to answer. If you'd asked me five years ago, I think I would have been more refined in my answer and I would have said, no, that's not a slippery slope problem. There's a slippery slope between I collect five measures and report one versus I collect 10 measures and I report one.

That's slippery slope. But making up data feels qualitatively different. And I still largely stand by that view. But there have been enough anecdotes that other people, whistleblower types, have presented to us that sound a lot more like someone says, yeah, you know, at first you do the thing where you drop some measures or drop a condition or you remove the outliers.

That's slippery slope. But making up data feels qualitatively different. And I still largely stand by that view. But there have been enough anecdotes that other people, whistleblower types, have presented to us that sound a lot more like someone says, yeah, you know, at first you do the thing where you drop some measures or drop a condition or you remove the outliers.

And then also you take participant 35 and you change their answer from a 7 to a 9. You're like, whoa, that last one doesn't sound the same. But maybe there's some psychology for that, that it feels like it's an extension.

And then also you take participant 35 and you change their answer from a 7 to a 9. You're like, whoa, that last one doesn't sound the same. But maybe there's some psychology for that, that it feels like it's an extension.

The very first blog post we posted was about identification of fraudulent data in a paper published 10 years ago. And that one was discovered because Yuri had made a chart for a totally different paper where he was mining data from multiple published studies to just make a chart. And I looked at his figure of this other research group's data and said, that seems unusual.

The very first blog post we posted was about identification of fraudulent data in a paper published 10 years ago. And that one was discovered because Yuri had made a chart for a totally different paper where he was mining data from multiple published studies to just make a chart. And I looked at his figure of this other research group's data and said, that seems unusual.

I want to go read that paper. And so I read the paper and then looked at that data set. In that one, it had collected data on a nine-point interval scale, so people can answer one, two, three, up through nine. And there were numbers in the data set that were things like negative 1.7. And so you say, oh, okay, we're done. Nothing fancy.

I want to go read that paper. And so I read the paper and then looked at that data set. In that one, it had collected data on a nine-point interval scale, so people can answer one, two, three, up through nine. And there were numbers in the data set that were things like negative 1.7. And so you say, oh, okay, we're done. Nothing fancy.

Once you open the data set, you can then close it and say it's broken.

Once you open the data set, you can then close it and say it's broken.

Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud.

Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud.

Bridging the gap between those two positions is this other entity, Harvard University. We only know what they've said outwardly, which is that they've put her on administrative leave, and they've recommended the retraction of those four papers, or the retraction of three plus an amendment to a previously retracted paper.

Bridging the gap between those two positions is this other entity, Harvard University. We only know what they've said outwardly, which is that they've put her on administrative leave, and they've recommended the retraction of those four papers, or the retraction of three plus an amendment to a previously retracted paper.

Certainly scary. Scary because it's so unfamiliar. I found out from talking, basically I was exchanging emails with a reporter. And so between these emails, she came back to me and was like, well, now given the lawsuit, would you like to add a new comment? And I was basically like, what, what lawsuit are you talking about?

Certainly scary. Scary because it's so unfamiliar. I found out from talking, basically I was exchanging emails with a reporter. And so between these emails, she came back to me and was like, well, now given the lawsuit, would you like to add a new comment? And I was basically like, what, what lawsuit are you talking about?

And so it's like this devastating thing to be like, oh my God, it's like the whole house is collapsing and no one told me.

And so it's like this devastating thing to be like, oh my God, it's like the whole house is collapsing and no one told me.

She was at the center of everything. Being a prestigious faculty member at Harvard and all of her public speaking and her books. Her reputation was perfect. She was synonymous with the highest levels of research in organizational behavior. She's just a giant in the field.

She was at the center of everything. Being a prestigious faculty member at Harvard and all of her public speaking and her books. Her reputation was perfect. She was synonymous with the highest levels of research in organizational behavior. She's just a giant in the field.

It was a tense few months, but in the end, I was allowed to continue doing what I was doing.

It was a tense few months, but in the end, I was allowed to continue doing what I was doing.

So I think journals have really complicated incentives.

So I think journals have really complicated incentives.

Of course, they want to publish good work to begin with, so there's some incentive to do some quality check and kind of cover their ass there. But once they've published something, there's a strong incentive for them to defend it or at least to not publicize any errors.

Of course, they want to publish good work to begin with, so there's some incentive to do some quality check and kind of cover their ass there. But once they've published something, there's a strong incentive for them to defend it or at least to not publicize any errors.

So one of the things the editor-in-chief does is when a manuscript is submitted, I would read it and decide whether it should continue through the peer review process or I could reject it there. And that's called desk rejection.

So one of the things the editor-in-chief does is when a manuscript is submitted, I would read it and decide whether it should continue through the peer review process or I could reject it there. And that's called desk rejection.

One thing I started doing at the journal that wasn't official policy, it was just a practice I decided to adopt, was that when a manuscript was submitted, I would hide the author's names from myself. So I was rejecting things without looking at who the authors were. So the publication committee started a conversation with me, which is totally reasonable, about the overall desk rejection rate.

One thing I started doing at the journal that wasn't official policy, it was just a practice I decided to adopt, was that when a manuscript was submitted, I would hide the author's names from myself. So I was rejecting things without looking at who the authors were. So the publication committee started a conversation with me, which is totally reasonable, about the overall desk rejection rate.

Am I rejecting too many things, etc.? There was some conversation about whether I was desk rejecting the wrong people. So if I was stepping on important people's toes and an email was forwarded to me from a quote unquote award winning social psychologist, you know, Samin desk rejected my paper. I found this extremely distasteful and I won't be submitting there again.

Am I rejecting too many things, etc.? There was some conversation about whether I was desk rejecting the wrong people. So if I was stepping on important people's toes and an email was forwarded to me from a quote unquote award winning social psychologist, you know, Samin desk rejected my paper. I found this extremely distasteful and I won't be submitting there again.

And when I would try to engage about the substance of my decisions, you know, the scientific basis for them, that wasn't what the conversation was about.

And when I would try to engage about the substance of my decisions, you know, the scientific basis for them, that wasn't what the conversation was about.

Yeah, yeah.

Yeah, yeah.

It was a tense few months, but in the end, I was allowed to continue doing what I was doing.

It was a tense few months, but in the end, I was allowed to continue doing what I was doing.

We're expanding a team that used to have a different name. We're going to call them the Statistics, Transparency and Rigor editors, the star editors. And so that team will be supplementing the handling editors, the editors who actually organize the peer review and make the decisions on submissions.

We're expanding a team that used to have a different name. We're going to call them the Statistics, Transparency and Rigor editors, the star editors. And so that team will be supplementing the handling editors, the editors who actually organize the peer review and make the decisions on submissions.

Like if a handling editor has a question about the data integrity or about details of the methods or things like that, the star editor team will provide their expertise and help fill in those gaps. We're also, I'm not sure exactly what form this will take, but try to incentivize more accurate and calibrated claims and less hype and exaggeration.

Like if a handling editor has a question about the data integrity or about details of the methods or things like that, the star editor team will provide their expertise and help fill in those gaps. We're also, I'm not sure exactly what form this will take, but try to incentivize more accurate and calibrated claims and less hype and exaggeration.

This is something that I think is particularly challenging with short articles like psychological science publishes and especially, you know, a journal that has really high rejection rate where the vast majority of submissions are rejected. authors are competing for those few spots. And so it feels like they have to make a really bold claim.

This is something that I think is particularly challenging with short articles like psychological science publishes and especially, you know, a journal that has really high rejection rate where the vast majority of submissions are rejected. authors are competing for those few spots. And so it feels like they have to make a really bold claim.

And so it's going to be very difficult to play this like back and forth where authors are responding to the perception of what the incentives are. So we need to convey to them that actually, if you go too far, make too bold of claims that aren't warranted, you will be more likely to get rejected.

And so it's going to be very difficult to play this like back and forth where authors are responding to the perception of what the incentives are. So we need to convey to them that actually, if you go too far, make too bold of claims that aren't warranted, you will be more likely to get rejected.

But I'm not sure if authors will believe that just because we say that they're still competing for a very selective number of spots.

But I'm not sure if authors will believe that just because we say that they're still competing for a very selective number of spots.

Oh, I don't mind being wrong. I think journalists should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we're only going to publish true things or things that we're 100% sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we'll make mistakes even there.

Oh, I don't mind being wrong. I think journalists should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we're only going to publish true things or things that we're 100% sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we'll make mistakes even there.

Sometimes we'll present things with certainty that we shouldn't have presented. What I would like to be involved in and what I plan to do is to encourage more post-publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon, and create more incentives for people to do that and do that well.

Sometimes we'll present things with certainty that we shouldn't have presented. What I would like to be involved in and what I plan to do is to encourage more post-publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon, and create more incentives for people to do that and do that well.

I don't know. Do you have any ideas? No.

I don't know. Do you have any ideas? No.

Stephen, I'm the person that walks into these academic conferences and everyone is like, here comes Debbie Downer.

Stephen, I'm the person that walks into these academic conferences and everyone is like, here comes Debbie Downer.

Bad question? No. Like, it reminds me of how stressful it all is. We struggle a little bit with thinking about analogies for what we do. We're definitely not police. Police, amongst other things, have institutional power. They have badges, whatever. We don't have any of that. We're not enforcers in any way. The internal affairs thing...

Bad question? No. Like, it reminds me of how stressful it all is. We struggle a little bit with thinking about analogies for what we do. We're definitely not police. Police, amongst other things, have institutional power. They have badges, whatever. We don't have any of that. We're not enforcers in any way. The internal affairs thing...

Hurts a little bit, but I get it, because that's saying, hey, within the behavioral science community, we're the people that are watching the behavioral scientists. And you're right, no one likes internal affairs. Most of our thinking is that we want to be journalists, that it's fun to investigate. That's true for everybody in the field, right?

Hurts a little bit, but I get it, because that's saying, hey, within the behavioral science community, we're the people that are watching the behavioral scientists. And you're right, no one likes internal affairs. Most of our thinking is that we want to be journalists, that it's fun to investigate. That's true for everybody in the field, right?

They're all curious about whatever it is they're studying. And so we're curious about this. And then when we find things that we think are interesting, we also want to talk about it, not just with each other, but with the outside world.

They're all curious about whatever it is they're studying. And so we're curious about this. And then when we find things that we think are interesting, we also want to talk about it, not just with each other, but with the outside world.

But I don't identify as much with being a police officer or even a detective, though every now and then people will compare us to something like Sherlock Holmes, and that feels more fun. But in truth, the reason I sort of wince at the question is that the vast majority of the time, it comes with far more burden than it does pleasure.

But I don't identify as much with being a police officer or even a detective, though every now and then people will compare us to something like Sherlock Holmes, and that feels more fun. But in truth, the reason I sort of wince at the question is that the vast majority of the time, it comes with far more burden than it does pleasure.

Yeah. The lawsuit makes all of the psychological burden into a concrete, observable thing. But the part prior to that is that every time we report on anything... That's going to be like, look, we think something bad happened here. Someone is going to be mad at us. And probably more people are going to be. And I don't want people to be mad at me. And I think about some of the people involved.

Yeah. The lawsuit makes all of the psychological burden into a concrete, observable thing. But the part prior to that is that every time we report on anything... That's going to be like, look, we think something bad happened here. Someone is going to be mad at us. And probably more people are going to be. And I don't want people to be mad at me. And I think about some of the people involved.

And it's hard because I know a lot of these people and I know their friends and I know the friends of the friends. And that carries real, real stress for, I think, all three of us.

And it's hard because I know a lot of these people and I know their friends and I know the friends of the friends. And that carries real, real stress for, I think, all three of us.

Yeah, that's real bad. I'm not happy with being compared to the Stassi. The optimistic take is that there's less of that than there used to be.

Yeah, that's real bad. I'm not happy with being compared to the Stassi. The optimistic take is that there's less of that than there used to be.

When any of the three of us go and visit universities, for example, and we talk to doctoral students and we talk to assistant professors and we talk to associate professors, we talk to senior professors, the students basically all behave as though they don't understand why anyone would ever be against what we're saying.

When any of the three of us go and visit universities, for example, and we talk to doctoral students and we talk to assistant professors and we talk to associate professors, we talk to senior professors, the students basically all behave as though they don't understand why anyone would ever be against what we're saying.

They wouldn't understand the Stasi thing, but they also wouldn't even understand like why they almost are at the level, I don't understand why we're having you come for a talk. Doesn't everyone already believe this? But when I talk to people that are closer to retirement than they are to being a grad student, they're more like, you know, you're making waves where you don't need to.

They wouldn't understand the Stasi thing, but they also wouldn't even understand like why they almost are at the level, I don't understand why we're having you come for a talk. Doesn't everyone already believe this? But when I talk to people that are closer to retirement than they are to being a grad student, they're more like, you know, you're making waves where you don't need to.

You're pushing back against something that's not there. We've been doing this for decades. Why fix what isn't broken? That sort of thing.

You're pushing back against something that's not there. We've been doing this for decades. Why fix what isn't broken? That sort of thing.

I would say, but it is broken. And your evidence for that would be? The evidence for that is multifold.

I would say, but it is broken. And your evidence for that would be? The evidence for that is multifold.

Sometimes, for example, we'll get a submission where the research is really solid, but the conclusion is too strong. And I'll sometimes tell authors, hey, look, I'll publish your paper if you tone down the conclusion or even sometimes change the conclusion from saying there is evidence for my hypothesis to there's no evidence one way or the other, but it's still interesting data.

Sometimes, for example, we'll get a submission where the research is really solid, but the conclusion is too strong. And I'll sometimes tell authors, hey, look, I'll publish your paper if you tone down the conclusion or even sometimes change the conclusion from saying there is evidence for my hypothesis to there's no evidence one way or the other, but it's still interesting data.

And authors are not always willing to do that, even if it means getting a publication in this journal. So I do think that's a sign that maybe it's a sign that they genuinely believe what they're saying, which is maybe to their credit. I don't know if that's good news or bad news. I think often when we're kind of overselling something, we probably believe what we're saying.

And authors are not always willing to do that, even if it means getting a publication in this journal. So I do think that's a sign that maybe it's a sign that they genuinely believe what they're saying, which is maybe to their credit. I don't know if that's good news or bad news. I think often when we're kind of overselling something, we probably believe what we're saying.

Editors largely in my field are uncompensated for their job, and reviewers are almost purely uncompensated for their job. And so they're all doing it for the love of the field. And those jobs are hard. I'm an occasional reviewer and an occasional editor. And every time I do it, it's basically taxing.

Editors largely in my field are uncompensated for their job, and reviewers are almost purely uncompensated for their job. And so they're all doing it for the love of the field. And those jobs are hard. I'm an occasional reviewer and an occasional editor. And every time I do it, it's basically taxing.

The first part of the job was reading a whole paper and deciding whether the topic was interesting, whether it was contextualized well enough that people would understand what it was about. Whether the study as designed was good at testing the hypothesis as articulated. And only after you get past all of those levels would you say, okay, and now do they have evidence in favor of the hypothesis?

The first part of the job was reading a whole paper and deciding whether the topic was interesting, whether it was contextualized well enough that people would understand what it was about. Whether the study as designed was good at testing the hypothesis as articulated. And only after you get past all of those levels would you say, okay, and now do they have evidence in favor of the hypothesis?

There were a lot of societal phenomena that we really wanted explanations for, and then social psych offered these kind of easy explanations, or maybe not so easy, but these relatively simple explanations that people wanted to believe just to have an answer and an explanation.

There were a lot of societal phenomena that we really wanted explanations for, and then social psych offered these kind of easy explanations, or maybe not so easy, but these relatively simple explanations that people wanted to believe just to have an answer and an explanation.

If you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat.

If you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat.

It was a tense few months, but in the end, I was allowed to continue doing what I was doing.

So I think journals have really complicated incentives.

Of course, they want to publish good work to begin with, so there's some incentive to do some quality check and kind of cover their ass there. But once they've published something, there's a strong incentive for them to defend it or at least to not publicize any errors.

So one of the things the editor-in-chief does is when a manuscript is submitted, I would read it and decide whether it should continue through the peer review process or I could reject it there. And that's called desk rejection.

One thing I started doing at the journal that wasn't official policy, it was just a practice I decided to adopt, was that when a manuscript was submitted, I would hide the author's names from myself. So I was rejecting things without looking at who the authors were. So the publication committee started a conversation with me, which is totally reasonable, about the overall desk rejection rate.

Am I rejecting too many things, etc.? There was some conversation about whether I was desk rejecting the wrong people. So if I was stepping on important people's toes and an email was forwarded to me from a quote unquote award winning social psychologist, you know, Samin desk rejected my paper. I found this extremely distasteful and I won't be submitting there again.

And when I would try to engage about the substance of my decisions, you know, the scientific basis for them, that wasn't what the conversation was about.

Yeah, yeah.

It was a tense few months, but in the end, I was allowed to continue doing what I was doing.

We're expanding a team that used to have a different name. We're going to call them the Statistics, Transparency and Rigor editors, the star editors. And so that team will be supplementing the handling editors, the editors who actually organize the peer review and make the decisions on submissions.

Like if a handling editor has a question about the data integrity or about details of the methods or things like that, the star editor team will provide their expertise and help fill in those gaps. We're also, I'm not sure exactly what form this will take, but try to incentivize more accurate and calibrated claims and less hype and exaggeration.

This is something that I think is particularly challenging with short articles like psychological science publishes and especially, you know, a journal that has really high rejection rate where the vast majority of submissions are rejected. authors are competing for those few spots. And so it feels like they have to make a really bold claim.

And so it's going to be very difficult to play this like back and forth where authors are responding to the perception of what the incentives are. So we need to convey to them that actually, if you go too far, make too bold of claims that aren't warranted, you will be more likely to get rejected.

But I'm not sure if authors will believe that just because we say that they're still competing for a very selective number of spots.

Oh, I don't mind being wrong. I think journalists should publish things that turn out to be wrong. It would be a bad thing to approach journal editing by saying we're only going to publish true things or things that we're 100% sure are true. The important thing is that the things that are more likely to be wrong are presented in a more uncertain way. And sometimes we'll make mistakes even there.

Sometimes we'll present things with certainty that we shouldn't have presented. What I would like to be involved in and what I plan to do is to encourage more post-publication critique and correction, reward the whistleblowers who identify errors that are valid and that need to be acted upon, and create more incentives for people to do that and do that well.

I don't know. Do you have any ideas? No.

Stephen, I'm the person that walks into these academic conferences and everyone is like, here comes Debbie Downer.

Bad question? No. Like, it reminds me of how stressful it all is. We struggle a little bit with thinking about analogies for what we do. We're definitely not police. Police, amongst other things, have institutional power. They have badges, whatever. We don't have any of that. We're not enforcers in any way. The internal affairs thing...

Hurts a little bit, but I get it, because that's saying, hey, within the behavioral science community, we're the people that are watching the behavioral scientists. And you're right, no one likes internal affairs. Most of our thinking is that we want to be journalists, that it's fun to investigate. That's true for everybody in the field, right?

They're all curious about whatever it is they're studying. And so we're curious about this. And then when we find things that we think are interesting, we also want to talk about it, not just with each other, but with the outside world.

But I don't identify as much with being a police officer or even a detective, though every now and then people will compare us to something like Sherlock Holmes, and that feels more fun. But in truth, the reason I sort of wince at the question is that the vast majority of the time, it comes with far more burden than it does pleasure.

Yeah. The lawsuit makes all of the psychological burden into a concrete, observable thing. But the part prior to that is that every time we report on anything... That's going to be like, look, we think something bad happened here. Someone is going to be mad at us. And probably more people are going to be. And I don't want people to be mad at me. And I think about some of the people involved.

And it's hard because I know a lot of these people and I know their friends and I know the friends of the friends. And that carries real, real stress for, I think, all three of us.

Yeah, that's real bad. I'm not happy with being compared to the Stassi. The optimistic take is that there's less of that than there used to be.

When any of the three of us go and visit universities, for example, and we talk to doctoral students and we talk to assistant professors and we talk to associate professors, we talk to senior professors, the students basically all behave as though they don't understand why anyone would ever be against what we're saying.

They wouldn't understand the Stasi thing, but they also wouldn't even understand like why they almost are at the level, I don't understand why we're having you come for a talk. Doesn't everyone already believe this? But when I talk to people that are closer to retirement than they are to being a grad student, they're more like, you know, you're making waves where you don't need to.

You're pushing back against something that's not there. We've been doing this for decades. Why fix what isn't broken? That sort of thing.

I would say, but it is broken. And your evidence for that would be? The evidence for that is multifold.

Sometimes, for example, we'll get a submission where the research is really solid, but the conclusion is too strong. And I'll sometimes tell authors, hey, look, I'll publish your paper if you tone down the conclusion or even sometimes change the conclusion from saying there is evidence for my hypothesis to there's no evidence one way or the other, but it's still interesting data.

And authors are not always willing to do that, even if it means getting a publication in this journal. So I do think that's a sign that maybe it's a sign that they genuinely believe what they're saying, which is maybe to their credit. I don't know if that's good news or bad news. I think often when we're kind of overselling something, we probably believe what we're saying.

Editors largely in my field are uncompensated for their job, and reviewers are almost purely uncompensated for their job. And so they're all doing it for the love of the field. And those jobs are hard. I'm an occasional reviewer and an occasional editor. And every time I do it, it's basically taxing.

The first part of the job was reading a whole paper and deciding whether the topic was interesting, whether it was contextualized well enough that people would understand what it was about. Whether the study as designed was good at testing the hypothesis as articulated. And only after you get past all of those levels would you say, okay, and now do they have evidence in favor of the hypothesis?

There were a lot of societal phenomena that we really wanted explanations for, and then social psych offered these kind of easy explanations, or maybe not so easy, but these relatively simple explanations that people wanted to believe just to have an answer and an explanation.

If you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat.

My name is Leif Nelson, and I'm a professor of business administration at University of California, Berkeley.

We started our blog in late 2013. We decided we wanted to have a blog because we thought it would be fun to write things that were shorter than a journal article and that we did not have to wait two and a half years for the review process to play out. And so with that in mind, we just needed to name it and we wanted something that would be related to what we do. That's maybe the data part.

but would definitely not be sending signals of self-seriousness. So we tried out a few things, and somewhere in there, De De Colato was one that we obviously landed on. It had this nice entertaining feature that Yuri is Chilean. And so when he had suggested the name... He thought it rhymed, which still tickles me and Joe, because for him it's data colada.

The classic forms of what we'd characterized as p-hacking, they're not quite errors, they're decisions that are accidentally self-serving. It's if you measure multiple things but only report the one you like the most.

Or you run a study where there's three treatments, condition A, condition B, and condition C, but in the end you drop condition B and you don't even talk about it, you just compare A to C.

And then there are things that are mildly statistical, but in a very relaxed way. Well, we collected this data, but it's kind of skewed. It has some outliers. and you say, we should eliminate those outliers, or we should Windsorize the outliers, which is basically truncating them down to a lower high number.

Or you could run them through an algorithm where you say, oh, let's transform them with a logarithm or with a square root. And those are all decisions that are justifiable. They're not crazy. It's just, if you have a consideration of reporting one variable or the other,

and one variable makes your hypothesis look good, and the other variable makes your hypothesis look less good, you end up reporting the one that looks good, either because you're being self-serving, or honestly, because you'd say, like, I'm not sure which one is better, but my hypothesis tells me it should be the one that looks good, and that one looks good. It's probably the better measure.

And here's Simonson.

These will be things that can be as simple as a typo. where someone's writing up their report and the means are actually 5.1 and 5.12, but instead someone writes it down as 51.2. And you're like, wow, that's a huge effect, right? And no one corrects it because it's a huge effect in the direction that they were expecting. And so literally a typo might end up in print.

And that's before we get to anything like fraud, like the active fabrication of data or manipulation of data.

Yeah. Well, Stephen, you were asking a question that is pretty heavy and one that I'm not particularly well-equipped to answer. If you'd asked me five years ago, I think I would have been more refined in my answer and I would have said, no, that's not a slippery slope problem. There's a slippery slope between I collect five measures and report one versus I collect 10 measures and I report one.

That's slippery slope. But making up data feels qualitatively different. And I still largely stand by that view. But there have been enough anecdotes that other people, whistleblower types, have presented to us that sound a lot more like someone says, yeah, you know, at first you do the thing where you drop some measures or drop a condition or you remove the outliers.

And then also you take participant 35 and you change their answer from a 7 to a 9. You're like, whoa, that last one doesn't sound the same. But maybe there's some psychology for that, that it feels like it's an extension.

The very first blog post we posted was about identification of fraudulent data in a paper published 10 years ago. And that one was discovered because Yuri had made a chart for a totally different paper where he was mining data from multiple published studies to just make a chart. And I looked at his figure of this other research group's data and said, that seems unusual.

I want to go read that paper. And so I read the paper and then looked at that data set. In that one, it had collected data on a nine-point interval scale, so people can answer one, two, three, up through nine. And there were numbers in the data set that were things like negative 1.7. And so you say, oh, okay, we're done. Nothing fancy.

Once you open the data set, you can then close it and say it's broken.

Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud.

Bridging the gap between those two positions is this other entity, Harvard University. We only know what they've said outwardly, which is that they've put her on administrative leave, and they've recommended the retraction of those four papers, or the retraction of three plus an amendment to a previously retracted paper.

Certainly scary. Scary because it's so unfamiliar. I found out from talking, basically I was exchanging emails with a reporter. And so between these emails, she came back to me and was like, well, now given the lawsuit, would you like to add a new comment? And I was basically like, what, what lawsuit are you talking about?

And so it's like this devastating thing to be like, oh my God, it's like the whole house is collapsing and no one told me.

She was at the center of everything. Being a prestigious faculty member at Harvard and all of her public speaking and her books. Her reputation was perfect. She was synonymous with the highest levels of research in organizational behavior. She's just a giant in the field.