Brian Nosek
Appearances
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We feel like when we're in cultures that there is no way for any of us to change the culture. It's a culture. My God, how could we change it? But we also recognize that cultures are created by the people that comprise them. And the notion that we collectively can actually do something to shift the research culture, I think, has spread.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And that spreading has actually accelerated the change of the research culture for the better.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, it's based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will disadvantage my career.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
I asked Nosik how he thinks this culture can be changed. So, for example, we have to make it easy for researchers to be more transparent with their work. If it's really hard to share your data, then adding on that extra work is going to slow down my progress. We have to make it normative. People have to be able to see that others in their community are doing this. They're being more transparent.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
They're being more rigorous so that instead of us saying, oh, that's great ideals and nobody does it, you say, oh, there's somebody over there that's doing it. Oh, maybe I could do it too. We have to deal with the incentives. Is it actually relevant for my advancement in my career to be transparent, to be rigorous, to be reproducible? And then we have to address the policy framework.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
If it's not embedded in how it is that funders decide who to fund, institutions decide who to hire, journals to decide what to publish, then it's not going to be internally and completely embedded in the system.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, so the idea is you register your designs and you've made that commitment in advance. And then as you're carrying out the research, if things change along the way, which happens all the time, you can update that registration. You can say, here's what's changing. We didn't anticipate that going into this community was going to be so hard and here's how we had to adapt. That's fine.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
You should be able to change. You just have to be transparent about those changes so that the reader can evaluate. And then those data are timestamped together? Exactly. Yeah. You put your data and your materials. If you did a survey, you add the surveys. If you did behavioral tasks, you can add those.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So all of that stuff can be attached then to the registration so that you have a more comprehensive record of what it is you did.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It makes fraud more inconvenient. And that's actually a reasonable intervention. I don't think any intervention that we could design could prevent fraud in a way that doesn't stifle actual legitimate research. We just want to make visible all the things that legitimate researchers are doing so that someone that doesn't want to do that extra work has a harder time.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And eventually, if everything is exposed, then the person who would be motivated to do fraud might say, well, it's just as easy to do the research the real way. So I guess I'll do that.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So in the standard publishing model, I do all of my research. I get my findings. I write it up in a paper and I send it to the journal. In that model, the reward system is about the findings. I need to get those findings to be as positive, novel, and tidy as I can so that you, the reviewer, say, OK, OK, you can publish it.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
That's dysfunctional and it leads to all of those practices that might lead the claims to be more exaggerated than the evidence.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
The registered report model says to the journal, you are going to submit, Brian, the methodology that you're thinking about doing and why you're asking that question and the background research supporting that question being important and that methodology being effective methodology. We'll review that. We don't know what the results are. You don't know what the results are.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But we're going to review based on, do you have an important question?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Exactly. And the key part is that the reward, me getting that publication, is based on you agreeing that I'm asking an important question and I've designed an effective method to test it. It's no longer about the results. None of us know what the results are.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Yeah, so the commitment that the journal makes is we're going to publish it regardless of outcome, and the authors are making that commitment too. We're going to carry this out as we said we would, and we'll report what happens.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Now, an interesting thing happens in the change of the culture here in evaluating research because you said, well, if it's an uninteresting finding, do we still have to publish it? It turns out that when you have to make a decision of whether to publish or not before knowing that the results are –
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
The orientation that the reviewers bring, that the authors bring, is do we need to know the answer to this? Regardless of what happens, do we need to know the answer? Is the question important, in other words? Exactly. Is the question important enough that we need evidence, regardless of what the evidence is? And it dramatically shifts what ends up being published.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
So in the early evidence with registered reports, more than half of the hypotheses that are proposed end up not being supported in the final paper. In the standard literature, comparable type of domains, more than 95% of the hypotheses are supported in the paper. You wonder in the standard literature, if we're always right, why do we bother doing the research, right?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Our hypotheses are always right. And of course, it's laughable because we know that's not what's actually happening. We know that all that failed stuff is getting left out and we're not seeing it. And the actual literature is an exaggeration of what the real literature is.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
I think there is broad buy-in on the need to change, and it has already hit the mainstream of many of the changes that we promote, sharing data, materials, code, pre-registering research, reporting all outcomes. So we're in the scaling phase for those activities, and what I am optimistic about is that we
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
There is this meta-science community that is interrogating whether these solutions are actually having the desired impact. And so this is the most exciting part of the movement as I'm looking to the future is this dialogue between activism and reform. We can do these things. Let's make these changes. And meta-science and evaluation. Is this working? Did you do what you said it was going to do?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And et cetera. And I hope that the tightness of that loop will stay tight because that, I think, will make for a very healthy discipline that is constantly skeptical of itself and constantly looking to do better.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
There really is accelerating movement in the sense that some of the base principles of we need to be more transparent, we need to improve data sharing, we need to facilitate the processes of self-correction are not just head nods, yeah, that's an important thing, but have really moved into, yeah, how are we going to do that?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And so I guess that's been the theme of 2024 is how can we help people do it well?
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
One of the more exciting things that we've been working on is a new initiative that we're calling Lifecycle Journal. And the basic idea is to reimagine scholarly publishing without the original constraints of paper. A lot of how the peer review process and publishing occurs today was done because of the limits of paper.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
But in a world where we can actually communicate digitally, there's no reason that we need to wait till the research is done to provide some evaluation. There's no reason to consider it final when it could be easily revised and updated. There's no reason to think of review as a singular one set of activities that by three people who judge the entire thing.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And so we will have a full marketplace of evaluation services that are each evaluating the research in different ways. It'll happen across the research lifecycle from planning through completion. And researchers will always be able to update and revise when errors or corrections are needed.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
My whole life is about trying to promote transparent research practices, greater openness, trying to improve rigor and reproducibility. I am just as vulnerable to error as anybody else. And so one of the real lessons, I think, is that without transparency, these errors will go unexposed.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
It would have been very hard for the critics to identify that we had screwed this up without being able to access the portions of the materials that we were able to make public. And as people are engaged with critique and pursuing transparency – and transparency is becoming more normal –
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We might, for a while, see an ironic effect, which is transparency seems to be associated with poorer research because more errors are identified. And that ought to happen because errors are occurring. Without transparency, you can't possibly catch them. But what might emerge over time as our verification processes improve, as we have a sense of accountability to our transparency,
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
then the fact that transparency is there may decrease error over time, but not the need to check. And that's the key.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
This is a real challenge that we wrestle with and have wrestled with since the origins of the center is how do we promote this culture of critique and self-criticism about our field and and simultaneously have that be understood as the strength of research rather than its weakness.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
One of the phrases that I've liked to use in this is that the reason to trust science is because it doesn't trust itself. That part of what makes science great as a social system is its constant self-scrutiny and willingness to try to find and expose its errors So that the evidence that comes out at the end is the most robust, reliable, valid evidence as could be.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
And that continuous process is the best process in the world that we've ever invented for knowledge production. We can do better. I think our mistake in some prior efforts of promoting science is to appeal to authority, saying you should trust science because scientists know what they're doing. I don't think that's the way to gain trust in science because anyone can make that claim.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
Appeals to authority are very weak arguments. I think our opportunity as a field to address the skepticism of institutions generally and science specifically is is to show our work, is by being transparent, by allowing the criticism to occur, by in fact encouraging and promoting critical engagement with our evidence.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
That is the playing field I'd much rather be on with people who are the so-called enemies of science than in competing appeals to authority. Because if they need to wrestle with the evidence and an observer says, wow, one group is totally avoiding the evidence and the other group is actually showing their work, I think people will know who to trust. That's easy to say. It's very hard to do.
Freakonomics Radio
Can Academic Fraud Be Stopped? (Update)
We are a carrot-based organization because we don't have sticks. I mean, would you like me to loan you a stick just once in a while? Yeah, that would be fun.