Brian Nosek
Appearances
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
We can't say with any confidence where it's most prominent. We can only say that the incentives for doing it are everywhere. And some of them gain more attention because, for example, Francesca's findings are interesting. They're interesting to everyone. So of course, they're going to get some attention to that. Whereas the anesthesiologist's findings are not interesting. They put people to sleep.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Until they kill you. Well, yeah, I guess they put you, and then they kill you.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Yeah, I would say that there is a lot of attention to social psychology for two reasons. One is that it has public engagement interest value. That's a way of saying that people care about your findings. people care about, at least in the sense of, oh, that's interesting to learn, right? But the other reason is that social psychology has bothered to look.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And I think social psychology became a hotbed for this because the actual challenges in the social system of science that need to be addressed are social psychological problems. What do you mean by that? I mean like the reward system, how it is that people might rationalize and use motivated reasoning to come to findings that are less credible.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
A lot of these problems are ones that social psychologists spend every day thinking about.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
The case that people make to say that this is a bigger problem now is that the competition for attention, jobs, advancement is very high, perhaps greater than it's ever been.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Yeah. So there are many more people and many fewer positions, which is an obvious challenge for a competitive marketplace. And there are now pathways for public attention that have even bigger impact. Academics, by and large, didn't think about ways to get rich. They looked for ways to have time to think about the problems that they want to think about. But now they have pathways to get rich.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Fraud has existed since science has existed. And that's primarily because humans are doing the science. And people come with ideas, beliefs, motivations, reasons that they're doing the research that they do. And in some cases, people are so motivated to advance an idea or themselves that they are willing to change the evidence fraudulently to advance that idea or themselves.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Our funders include NIH, NSF, NASA, and DARPA as federal sources, and then a variety of private sources such as the John Templeton Foundation, Arnold Foundation, and many others. And that diverse group of funders – and it's quite diverse – I think share the recognition that the substantive things that they are trying to solve –
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
won't be solved very effectively if the work itself is not done credibly.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
There are specific cases where a finding gets translated into public policy or into some type of activity that then ends up actually damaging people, lives, treatments, solutions. One of the most prominent examples is the Wakefield scandal relating to development of autism and the notion that vaccines might contribute to
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And that has had an incredibly corrosive impact on public health, on people's beliefs about the sources of autism, the impacts of vaccines, et cetera. And that is very costly for the world. There's also a local cost in academic research, which is just a ton of waste.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
So even if it doesn't have public downstream consequences, if a false idea is in the literature and other people are trying to build on it, it's just waste, waste, waste, waste.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Yeah, I have always had an interest in how to do good science as a principled matter. And in doing that, we in the lab would work on developing tools and resources to be more transparent with our work, to try to be more rigorous with our work, to try to do higher powered research. more sensitive research designs.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And so I wrote grant applications to say, can we make a repository where people can share their data? This is like 2007. And they would get polarized reviews where some reviewers would say, this would change everything. It'd be so useful to be more transparent with our work. And others saying, but researchers don't like sharing their data. Why would we do that?
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And why would researchers not want to share their data? Yeah, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
I have a very strong sense of, oh, my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will disadvantage my career.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Yeah. And here's the irony is that almost every academic would say, of course, science is supposed to be transparent. Of course, we're doing research for the public good. Of course, this is all to be shared. But come on, Pollyanna, we live in a world, right? The reality here is that there is a reward system and I have to have a career in order to do that research.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And so, yes, we can talk all about those ideals of transparency and sharing and rigor, reproducibility. But if they're not part of the reward system, you're asking me to either behave by my ideals and not have a career or have a career and sacrifice some of those ideals.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And in the end of that, 2015, we published The Findings, which was a 270 co-author paper of 100 replications of findings from three different journals in psychology. We got a little less than half of the findings successfully replicated.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Little less than half of the findings successfully replicated.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
A year and a half ago, we published the results of the reproducibility project in cancer biology doing the same kind of process and found very similar results. Less than half of the findings in preclinical cancer research replicated successfully when we tried to do so.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
That doesn't mean that the original finding is necessarily wrong. We could have screwed something up in the replication. Successfully replicating doesn't mean the interpretation is right. It could be that both the findings have a confound in them, but we just are able to repeat the confound.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
Fraud is the ultimate corrosive element of the system of science because as much as transparency provides some replacement for trust – You can't be transparent about everything. So the ideal model in scholarship is that you can see how it is they generated their evidence, how they interpreted their evidence, what the evidence actually is, and then independent people can interrogate that.
Freakonomics Radio
Why Is There So Much Fraud in Academia? (Update)
And so to the extent that fraud intrudes and actually the evidence isn't It isn't actual evidence. Then the whole edifice of that scholarly debate and tangling with ideas falls apart because you're actually tangling with ideas that aren't based on anything. How familiar are you with the Joachim Bolt situation? I'm not recalling that name, but I may know the case if you describe it.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We feel like when we're in cultures that there is no way for any of us to change the culture. It's a culture. My God, how could we change it? But we also recognize that cultures are created by the people that comprise them. And the notion that we collectively can actually do something to shift the research culture, I think, has spread.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And that spreading has actually accelerated the change of the research culture for the better.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will disadvantage my career.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
I asked Nosik how he thinks this culture can be changed. So, for example, we have to make it easy for researchers to be more transparent with their work. If it's really hard to share your data, then adding on that extra work is going to slow down my progress. We have to make it normative. People have to be able to see that others in their community are doing this. They're being more transparent.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
They're being more rigorous so that instead of us saying, oh, that's great ideals and nobody does it, you say, oh, there's somebody over there that's doing it. Oh, maybe I could do it too. We have to deal with the incentives. Is it actually relevant for my advancement in my career to be transparent, to be rigorous, to be reproducible? And then we have to address the policy framework.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
If it's not embedded in how it is that funders decide who to fund, institutions decide who to hire, journals to decide what to publish, then it's not going to be internally and completely embedded in the system.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, so the idea is you register your designs and you've made that commitment in advance. And then as you're carrying out the research, if things change along the way, which happens all the time, you can update that registration. You can say, here's what's changing. We didn't anticipate that going into this community was going to be so hard and here's how we had to adapt. That's fine.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
You should be able to change. You just have to be transparent about those changes so that the reader can evaluate. And then those data are timestamped together? Exactly. Yeah. You put your data and your materials. If you did a survey, you add the surveys. If you did behavioral tasks, you can add those.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So all of that stuff can be attached then to the registration so that you have a more comprehensive record of what it is you did.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It makes fraud more inconvenient. And that's actually a reasonable intervention. I don't think any intervention that we could design could prevent fraud in a way that doesn't stifle actual legitimate research. We just want to make visible all the things that legitimate researchers are doing so that someone that doesn't want to do that extra work has a harder time.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And eventually, if everything is exposed, then the person who would be motivated to do fraud might say, well, it's just as easy to do the research the real way. So I guess I'll do that.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So in the standard publishing model, I do all of my research. I get my findings. I write it up in a paper and I send it to the journal. In that model, the reward system is about the findings. I need to get those findings to be as positive, novel, and tidy as I can so that you, the reviewer, say, OK, OK, you can publish it.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
That's dysfunctional and it leads to all of those practices that might lead the claims to be more exaggerated than the evidence.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
The registered report model says to the journal, you are going to submit, Brian, the methodology that you're thinking about doing and why you're asking that question and the background research supporting that question being important and that methodology being effective methodology. We'll review that. We don't know what the results are. You don't know what the results are.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But we're going to review based on, do you have an important question?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Exactly. And the key part is that the reward, me getting that publication, is based on you agreeing that I'm asking an important question and I've designed an effective method to test it. It's no longer about the results. None of us know what the results are.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, so the commitment that the journal makes is we're going to publish it regardless of outcome, and the authors are making that commitment too. We're going to carry this out as we said we would, and we'll report what happens.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Now, an interesting thing happens in the change of the culture here in evaluating research because you said, well, if it's an uninteresting finding, do we still have to publish it? It turns out that when you have to make a decision of whether to publish or not before knowing that the results are –
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
The orientation that the reviewers bring, that the authors bring, is do we need to know the answer to this? Regardless of what happens, do we need to know the answer? Is the question important, in other words? Exactly. Is the question important enough that we need evidence, regardless of what the evidence is? And it dramatically shifts what ends up being published.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So in the early evidence with registered reports, more than half of the hypotheses that are proposed end up not being supported in the final paper. In the standard literature, comparable type of domains, more than 95% of the hypotheses are supported in the paper. You wonder in the standard literature, if we're always right, why do we bother doing the research, right?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Our hypotheses are always right. And of course, it's laughable because we know that's not what's actually happening. We know that all that failed stuff is getting left out and we're not seeing it. And the actual literature is an exaggeration of what the real literature is.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
I think there is broad buy-in on the need to change, and it has already hit the mainstream of many of the changes that we promote, sharing data, materials, code, pre-registering research, reporting all outcomes. So we're in the scaling phase for those activities, and what I am optimistic about is that we
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
There is this meta-science community that is interrogating whether these solutions are actually having the desired impact. And so this is the most exciting part of the movement as I'm looking to the future is this dialogue between activism and reform. We can do these things. Let's make these changes. And meta-science and evaluation. Is this working? Did you do what you said it was going to do?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And et cetera. And I hope that the tightness of that loop will stay tight because that, I think, will make for a very healthy discipline that is constantly skeptical of itself and constantly looking to do better.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
There really is accelerating movement in the sense that some of the base principles of we need to be more transparent, we need to improve data sharing, we need to facilitate the processes of self-correction are not just head nods, yeah, that's an important thing, but have really moved into, yeah, how are we going to do that?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And so I guess that's been the theme of 2024 is how can we help people do it well?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
One of the more exciting things that we've been working on is a new initiative that we're calling Lifecycle Journal. And the basic idea is to reimagine scholarly publishing without the original constraints of paper. A lot of how the peer review process and publishing occurs today was done because of the limits of paper.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But in a world where we can actually communicate digitally, there's no reason that we need to wait till the research is done to provide some evaluation. There's no reason to consider it final when it could be easily revised and updated. There's no reason to think of review as a singular one set of activities that by three people who judge the entire thing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And so we will have a full marketplace of evaluation services that are each evaluating the research in different ways. It'll happen across the research lifecycle from planning through completion. And researchers will always be able to update and revise when errors or corrections are needed.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
My whole life is about trying to promote transparent research practices, greater openness, trying to improve rigor and reproducibility. I am just as vulnerable to error as anybody else. And so one of the real lessons, I think, is that without transparency, these errors will go unexposed.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It would have been very hard for the critics to identify that we had screwed this up without being able to access the portions of the materials that we were able to make public. And as people are engaged with critique and pursuing transparency – and transparency is becoming more normal –
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We might, for a while, see an ironic effect, which is transparency seems to be associated with poorer research because more errors are identified. And that ought to happen because errors are occurring. Without transparency, you can't possibly catch them. But what might emerge over time as our verification processes improve, as we have a sense of accountability to our transparency,
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
then the fact that transparency is there may decrease error over time, but not the need to check. And that's the key.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
This is a real challenge that we wrestle with and have wrestled with since the origins of the center is how do we promote this culture of critique and self-criticism about our field and and simultaneously have that be understood as the strength of research rather than its weakness.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
One of the phrases that I've liked to use in this is that the reason to trust science is because it doesn't trust itself. That part of what makes science great as a social system is its constant self-scrutiny and willingness to try to find and expose its errors So that the evidence that comes out at the end is the most robust, reliable, valid evidence as could be.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And that continuous process is the best process in the world that we've ever invented for knowledge production. We can do better. I think our mistake in some prior efforts of promoting science is to appeal to authority, saying you should trust science because scientists know what they're doing. I don't think that's the way to gain trust in science because anyone can make that claim.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Appeals to authority are very weak arguments. I think our opportunity as a field to address the skepticism of institutions generally and science specifically is is to show our work, is by being transparent, by allowing the criticism to occur, by in fact encouraging and promoting critical engagement with our evidence.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
That is the playing field I'd much rather be on with people who are the so-called enemies of science than in competing appeals to authority. Because if they need to wrestle with the evidence and an observer says, wow, one group is totally avoiding the evidence and the other group is actually showing their work, I think people will know who to trust. That's easy to say. It's very hard to do.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We are a carrot-based organization because we don't have sticks. I mean, would you like me to loan you a stick just once in a while? Yeah, that would be fun.