John List
Appearances
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Good job! She's smiling. Oh, that's great!
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
You can kind of put what we've learned into three general buckets that seem to encompass the failures. Bucket number one is that the evidence was just not there to justify scaling the program in the first place. The Department of Education did this broad survey on prevention programs attempting to attenuate youth substance and crime and aspects like that.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
And what they found is that only 8% of those programs were actually backed by research evidence. Many programs that we put in place really don't have the research findings to support them. And this is what a scientist would call a false positive. So are we talking about bad research? Are we talking about cherry picking? Are we talking about publication bias?
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
So here we're talking about none of those. We're talking about a small-scale research finding that was the truth in that finding But because of the mechanics of statistical inference, and it just won't be right, what you were getting into is what I would call the second bucket of why things fail, and that's what I call the wrong people were studied.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
You know, these are studies that have a particular sample of people that shows really large program effect sizes. But when your program is gone to general populations, that effect disappears. So essentially, we were looking at the wrong people and scaling to the wrong people.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
They are the people who are the fraction or the group of people who receive the largest program benefits.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
That's one piece of it. Another piece is... who will sign their kids up for Head Start or for a program in a neighborhood that advances the reading skills of the child? Who's going to be first in line? The people who really care about education and the people who think their child will receive the most benefits from the program.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Now, another way to get it is sort of along the lines that you talked about. It could be the researcher knows something about the population. That other people don't know. Like, I want to give my program its best shot of working. Okay. And what's in your third bucket of scaling failures? The third bucket is something that we call the wrong situation was used.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
And what I mean by that is that certain aspects of the situation change when you go from the original research to the scaled research program. we don't understand what properties of the situation or features of the environment will matter. There are a really large group of implementation scientists who have explored this question for years.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Now, what they emphasize and focus on is something called voltage drop. And voltage drop essentially means I found a really good result in my original research study. But then when they do it at scale, that voltage drop ends up being, for example, a tenth of the original result or a quarter of the original result.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
An example of this is when you look at Head Start's home visiting services, what they do there is this is an early childhood intervention.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
that found huge improvements in both child and parent outcomes in the original study, except when they tried to scale that up into home visits at a much larger scale, what they found is that, for example, home visits for at-risk families involved a lot more distractions in the house and there was less time on child-focused activities. So this is sort of
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
The wrong dosage or the wrong program is given at scale.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
When you think about the chef, if a restaurant succeeds because of the magical work of the chef, and you think about scaling that, if you can't scale the magic in the chef, that's not scalable. Now, if the magic is because of the mix of ingredients, And the secret sauce, like Domino's, for example, the secret sauce or Papa John's is the actual ingredients, then that will be scalable.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Now, our proposal is that we do not believe that we should scale a program until you're 95% certain the result is true. So essentially what that means is we need the original research and then three or four well-powered, independent replications of the original findings.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
My intuition is that they're probably not far away from three or four well-powered independent replications. In the hard sciences, in many cases, you not only have the original research, but you have a first replication also published in science. You know, the current credibility crisis in science is a serious one that major results are not replicating.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
The reason why is because we weren't serious about replication in the first place. So this sort of puts the onus on policymakers and funding agencies in a sense of saying, we need to change the equilibrium.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
That's John List. He's an economist at the University of Chicago. They actually have to go and get the medicines, which a lot of people have a very hard time doing, even though it's sitting next to your bed every night. People don't take it. And they don't take it because they forget. They don't take it because the side effect is a lot worse than the benefit they think they're getting.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Well, I think it's sort of a mix. I think it's fair to say that some policymakers are out looking for evidence to to base their preferred program on, what this will do is slow that down. If you have a pet project that you want to get through, fund the replications and let's make sure the science is correct. We think we should actually be rewarding scholars for attempting to replicate.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
You know, right now in my community, if I try to replicate someone else, guess what I've just made? I've just made a mortal enemy for life. If you find a publishable result, what result is that? You're refuting previous research. Now I've doubled down on my enemy. So that's like a first step in terms of rewarding scholars who are attempting to replicate.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Now, to complement that, I think we should also reward scholars who have produced results that are independently replicated. You know what I'm talking about? Tying tenure decisions, grant money, and the like to people who have given us credible research that replicates.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Say I'm doing an experiment in Chicago Heights on early childhood and I find a great result. How confident should I be that when we take that result to all of Illinois or all of the Midwest or all of America, is that result still going to find that important benefit cost profile that we found in Chicago Heights? We need to know what is the magic sauce.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Was it the 20 teachers you hired down in Chicago Heights where if we go nationally, we need 20,000? So it should behoove me as an original researcher to say, look, if this scales up, we're going to need many more teachers. I know teachers are an important input. Is the average teacher in the 20,000 the same as the average teacher in the 20?
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
And the implementation scientists have focused on fidelity as a core component behind the voltage drop.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
All of these types of problems, as humans, including myself, we do a really bad job in trying to solve.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
If you turn back the clock to the 1990s, there was a credibility revolution in economics, focusing on what data and modeling assumptions are necessary to go from correlation to causality. List responded by running dozens and dozens of field experiments.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
Now, my contribution in the credibility revolution was instead of working with secondary data, I actually went to the world and used the world as my lab and generated new data to test theories and estimate program effects.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
I think moving our work into policymaking circles and having a very strong impact has just not been there. And I think one of the most important questions is, how are we going to make that natural progression of field experiments within the social sciences to more keenly talk to policymakers, the broader public, and actually the scientific community as a whole?
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
In a past life, I worked in the White House advising the president on environmental and resource issues within economics.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
A harsh lesson that I learned was you have to evaluate the effects of public policy as opposed to its intentions.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
When you step back and look at the amount of policies that we put in place that don't work...
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
So down in Chicago Heights, I ran a series of interventions, and one of the more powerful interventions was called the Parent Academy. That was a program that brought in parents every few weeks, and we taught them everything.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
What are the best mechanisms and approaches that they can use with their three-, four-, and five-year-old children to push both their cognitive skills and their executive function skills, things like self-control? What we found was within three to six months, we can move a child in very short order to have very strong cognitive test scores and very strong executive function skills.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
So, of course, we're very optimistic after getting this type of result, and we want the whole world to now do parent academies. The UK approaches us and said, we want to roll it out across London and the boroughs around London. What we found is that it failed miserably. It wasn't that the program was bad. It failed miserably because no parents actually signed up.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
So if you want your program to work at higher levels, you have to figure out how to get the right people and all the people, of course, into the program.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
The main problem is we just don't understand the science of scaling.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
I do think it's a crisis in that if we don't take care of it as scientists, I think everything we do can be undermined in the eyes of the policymaker and the broader public. We don't understand how to use our own science to make better policies.
Freakonomics Radio
Policymaking Is Not a Science — Yet (Update)
So Dana and I met back in 2012. And we were introduced by a mutual friend. And we did the usual ignore each other for a few years because we're too busy. And push came to shove. Dana and I started to work on early childhood research. And after that, research turned to love.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Because in the past, it was move fast and break things, throw spaghetti against the wall, whatever sticks, cook it, fake it till you make it. And what I'm adding here is we need science to figure out which ideas are truly scalable.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Hey, thanks for having me.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So I started an early childhood program in Chicago Heights, whereby I had teachers teach three, four, and five-year-olds. And those three, four, and five-year-olds learned a lot within six or 12 months. Now, the problem with that program is when I try to scale it, when I originally did the program, I hired 30 really good teachers.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
But when I scale it up, I might have to hire 30,000 really good teachers. So that's a very difficult chore compared to when I hired 30. So that idea is not very scalable. What I should make sure to do in the original study is to have teachers who I can hire at scale and make sure my program works with those kinds of teachers. That's one way to think about scalability.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
would still work that's scalable i think that's right but it's also true that unique humans themselves don't scale and the reason why is because they're unique for a reason and it's difficult to teach other people to be unique so in a way if one of the important elements of your idea is something that you just can't get at scale It will never scale.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So let's take ourselves back to the polio vaccination and let's think what Jonas Salk did. What Jonas Salk did is he had an idea about a vaccination that could handle polio. So what he did is he tested it out on his own children to start. And then he tested again to make sure that the original results were not a false positive.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So he first of all figured out this thing works by testing and retesting. Then what he did is he tried it out on a lot of different kinds of children. So he found out what is the slice of the pie that my idea can work for? That was step two. And in step two, he found, wow, it works for all children. Then after he found out that it worked on all children, he went to step three.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And that's how do we actually get it in people's arms? But the brilliance behind the polio vaccination is that we leverage the healthcare system And what we essentially have done is after you have a child, as those listeners know, if you've had kids, baby comes out, gets whisked away and tested and get some vaccinations.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
You bring your baby back in six months, it gets more vaccinations, 12 months more, 18 months, a few more. So the polio vaccination is given in a natural way. Step four in my reasoning is, well, does your idea have spillover effects? And in this case, the idea of a polio vaccination has great spillover effects because once you get the vaccination, you can't pass along polio to other children.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So that's really good. And then the fifth hurdle that you have to jump over is how much does it actually cost you to provide it and get it in people's arms? And again, with the polio vaccination, the true expense came in the R and D of the actual product. And because we leverage the healthcare system, It's now a pretty inexpensive way to give to people.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So really the polio vaccination passes over these five hurdles of an idea that is perfectly scalable. And in a way it's worked brilliantly. And these features are what is really elemental signatures of ideas that scale.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
The DARE program was a social inoculation program that people basically said, don't use drugs. That original idea came from an experiment in Honolulu, Hawaii with 1,777 high schoolers that sent a signal that the idea works. But in the end, it was just a false positive. We ended up trying it in LA and we tried it in other cities and we tested it in those other cities and it didn't work.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
We then went back to Honolulu and tried it there and it doesn't work. So we know that because of science.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Well, it's changed. So the DARE program representatives read my book and they reached out to me after they read the book and they said, we don't like chapter one, basically. And I said, is chapter one incorrect? And they said, no, but the DARE program has pivoted and they've moved to a new type of program that they argue now works.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So I haven't looked at the new program, but I told them I will look at the new program. And if there's voltage there, then in the next edition, I'll make sure to talk about the new program as well. So I'm talking about the program in the 80s that was simply a false positive.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
The idea is not just about... growing really, really large and taking over the world. The idea is figuring out how big your idea can be. And then you have to determine whether you're happy with that. So think about my brother and dad and grandpa, one man, one truck, one good life. They realize their secret sauce is their charisma.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And they realize that they can have a really good life with one truck and them driving it. They realize that that's the extent that they want to scale. And they realize that their idea is not scalable. So they have to decide, is that the life that I want? Or should I try something new? And they're happy with that scalable idea.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So what I'm trying to make the point about is to be scalable, you have to have certain features. But if you don't have all of those features, you still might want to do the idea if that's good enough for you. And there are a lot of ideas that can't take over the world. But there are a lot of ideas that have scaled.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And you think about various products and ideas that work, they have the five features that I talk about. Ideas that might only have two or three of the features might scale, but the 10th is much smaller. So the idea is understand what are the signatures of ideas
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
an idea because in the past it was move fast and break things, throw spaghetti against the wall, whatever sticks, cook it, fake it till you make it. We've all heard these statements, but it's always art. And what I'm adding here is we need science to figure out which ideas are truly scalable.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So here's kind of what happened. There was an innovation of the smart thermostat. Put it in your house. It will moderate your energy use over 24 hour day. And the engineers estimated that we will have huge savings because of the smart thermostat. So here's what we did. We chose 400,000 households in California and we sent 200,000 of them the smart thermostat.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
In the other 200,000, we didn't send anything and we just observed all 400,000 households. Guess what happens? There is zero energy savings. Now you can say, well, wait a second. Why did the engineers so dramatically overestimate what would happen? Well, here's why. They assumed that the end user was Commander Spock. This is a guy who never makes a mistake, and he is 100% rational.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
That's not who the end users are of this product. The end users are more like Homer Simpson. Homer Simpson is exactly what I did. I got this new gadget. I did not read the 28 page manual, how to use it. And then I went in and fiddled with the new gadget and I undid all of the presets. I undid all of the defaults. That's what the 200,000 households did in California.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
They undid them exactly enough on average to undo all of the good stuff with the technology. That's knowing the situation. Know who's going to use your product and give them technology that they can actually use that will help save the Earth.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
That's why that idea did not scale from the Petri dish to the large, because we did not understand the end users are very different than Commander Spock from Star Trek.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And let me be clear, they're working for some people. So when I say on average, it doesn't work, that doesn't mean that it's not working for some people. And you're right. What happens now is you come to smart thermostat level two and then level three, and that product will evolve to be much more user-friendly. When we put, I used to be the chief economist at Lyft,
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And when we put out products there, whether it's called walk and save or wait and save, there's always some beta testing and then some evolution to make sure people understand the new product. The general idea here, though, is you might have a voltage drop. if your estimates from the beginning were that all the users are going to 100% understand and use a technology as you estimated to start out.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And in many cases, that doesn't happen. And that's just a general idea of a voltage drop.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
If you don't execute on the idea, if you're not a good manager and make good decisions, the best idea won't work. But on the other hand, if you're 100% at execution, if you're trying to scale an idea that doesn't have the good signatures, that won't scale either. So there are ideas all over the world that look good on paper, But because of bad execution, they just don't scale.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
many restaurants. When you think about Jamie Oliver, that was really bad execution. So Jamie's restaurants in the UK were really good early on and the execution ended up faltering. And there are many cases like that. I think Sears and Kmart were a bit like that too. Look at the 1955 Fortune 500 companies. In 1955, if you look at those 500 companies, only 70 of those are around today.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Now, in many cases, those are firms like Studebaker, Zenith, et cetera. They just did not pivot. And when the times caused you to change, they refuse to change. Blockbuster is much the same way. And in many cases, when markets change, if you're not pivoting, this is a lack of execution. The idea looked good early on, but then the market changes and you're not pivoting with it.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So the world is replete with examples now where Folks are not using data or not using new elements of data science to help make decisions. And I would say that those firms are endangered. Blockbuster could have purchased Netflix for pennies on the dollar had they wanted to, but they thought that's not the future.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
They could have easily pivoted in that direction and said, we're going to do both and we're going to be diversified, but they didn't. When you look at many firms, they start doing something very different than what they become.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Let's think about a pricing idea that I had, and this might be familiar to some of your listeners, and it's called left digit bias pricing. And the idea is that as humans, we engage in shortcuts. And when we see a number, we focus on the left most digit. So what does that mean? People open up their apps and say they get a price of $7.93.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
And then they make a decision whether to take the trip in Lyft or in an Uber. Now, that decision is not very different than if they receive a price of $7.94. Okay. Because humans see 794 as the same as 793. Now let's say I change that just a bit and I give you a price of 799 versus a price of $8.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
That's basically the same as the first example, but now because people are focusing on the seven or the eight, that penny difference makes a big difference in your decision. Okay, so I tested that idea using Lyft data and using Lyft, a big field experiment on Lyft. And I find that that one penny change makes a big difference. And I test it again and I find it's not a false positive. Okay.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
I then go to the next step to say, who does this work for? It ends up working for everyone. Everyone has this left digit bias. Okay. Now let's go to the situation. You have business travelers, people travel in morning rush hour, afternoon rush hour, going to the airport. It works in all of those situations.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So now that's great because I can control in a better way using pricing in this behavioral bias. So I can scale that up and I can effectively impact people's choices because of this simple left digit bias rule, scalable, It works. It's not costly to do it. Because remember, we have to give you a price anyway. What does it matter if I give you a price of $7.93, $7.99, or $8?
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So the supply side of it is great. And the spillovers are great too, because I can control whether you get it or not. So if I'm on the bad end of a market, what I mean by bad end is there are a lot more drivers than consumers. I can move consumption. Or if it's more consumers and drivers, I can move consumption.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
So now I can use this type of bias that humans have in a way to make the world a better place.
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Yeah, well, here's a hint. So... The main part of both of these algorithms are working on the demand and supply imbalance. So one company might have a lot more demand than they have drivers or vice versa. That's one reason. Another reason is these companies are always experimenting
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
with pricing so you might just be part of an experiment where you got the bad draw from one company and a good draw from the other one and by good draw i mean a lower price so it could be because of demand or supply or because they're experimenting with the prices but you're called a dual apper so a dual apper is always checking both and then taking which one is best
Something You Should Know
The New Science of Effective Weight & What Makes an Idea Scalable
Mike, thanks again. It was great chatting with you. I really love your curiosity, by the way.