October 23, 2013

Interview with John A. List, Homer J. Livingston Professor of Economics and Department of Economics Chairman, University of Chicago

Paula Tkac: Welcome to the Public Affairs Forum. I'm Paula Tkac, and I'm here today with John List from the University of Chicago to talk about his work with natural field experiments in trying to understand human motivation and behavior. Welcome.

John List: Thanks for having me.

Tkac: Tell me about the way that you approach the study of human behavior.

List: Instead of waiting to have the world give me interesting data sets, which is, as you say, exactly what most empiricists do, I actually go out to the real world and generate my own data. So, for example, I'm interested in why do people give to charity. I will actually partner with charities and have them run natural field experiments with me, and explore exactly why do people give to charities, what can we do to keep them committed to the cause, trying to figure out the underlying features of people's preferences of why do they actually give to charity.

Tkac: So are there particular questions that are really well suited for doing field experiments versus other kinds of economic work, or in reverse, are there some that you wouldn't want to touch with a field experiment?

List: Right. So let me give you a few examples of questions that I probably would not want to touch from a field experiment. One would be something along the lines of, "How does the price of tea in China affect female labor supply in the United States?" Now that's hard to answer because I can't control prices of tea in China. So that's a silly example, but what it gives you is an understanding that to run a field experiment you need control. You need to have control over some important economic variable that you can adjust, and then watch how people respond behaviorally.

Tkac: You were discussing your field experiments in the field of education, and you were talking about needing to make sure that all of the kids that you had signed up show up, and that it might be related to their parents' involvement. So maybe you can talk a little bit about either that example or another, but just what's important to make sure you adjust for a control and design so that you know your experiments are going to be good when you get them at the end.

List: That's a good point. I think one of the most important features that experimental work gives you is that you have a random selection of people who you put in the control group and a random selection of people who you put in the treatment group.

So let's talk about an example. Let's turn back the clock to the "Coleman Report." So in 1966, in education circles, there was a famous report published called the "Coleman Report." Now what that report said was that students in larger size classrooms, i.e., a student in a 20-student classroom, was actually better off in terms of academic growth than a student in a 15-student classroom. Now, sure enough, that's in the data.

Tkac: Seems a bit surprising.

List: Seems very surprising. Now at the time they justified it by saying that there was just more chance for a peer effect; that the more students you have in the classroom, the more peers you can draw from and you can learn from them. Now what actually happened was the econometrician did not realize that the classrooms that had more kids, they actually had better kids and better teachers. So the unobservable here that was causing selection problems was that the better teachers and the better kids were actually in larger size classrooms. And it was that variable that was causing those kids to grow more, it wasn't just the fact of having more kids.

Tkac: So now, I've done a few experiments myself in labs, not anything as exciting as what you do, but there was always attention, I felt, new to this field, in terms of thinking about what you can control in a lab that allows you to have, I think, a pretty good sense of what your subjects think and believe, and so then therefore you kind of feel as though you have a handle on how they're using information or why they're making a decision. But then you worry that you can't extrapolate that to the real world. And a field experiment, kind of, seems the reverse; you're in the real world but you may not have as much control. So I guess I'm curious from your perspective, how you think about that tension and whether you can address it or you just attempt to work around it.

List: Absolutely. So I think the biggest misunderstanding with experimental economics is control. So what is control? The way we learned about experimental economics as high schoolers is, we go into the lab and we're chemists and we need really, really clean test tubes. That's wrong...

Tkac: OK.

List: ... And here's why it's wrong. You can't get rid of dirtiness. The real world is complicated—many, many complex moving parts. The key that you need to understand is that you have to balance the dirt across the treatment and control groups. And once you do that, you don't need to get rid of it because once these unobservables are balanced, you can make proper inference. So all you need is randomization. Randomization balances the noise or the dirt. You don't get rid of it, you balance it.

Tkac: So tell me about your most surprising field experiment.

List: So let's choose one that might be thought-provoking and that's when we look at the gender pay gap. So a fact in the literature is that men earn about, let's say, 20 percent more than women. And then people want to find out exactly why do women earn 20 percent less than men? You have typical arguments such as women take more time out of the labor force. You have arguments that women have lower levels of human capital. You have arguments about discrimination. Now where we come at it, it's potentially preferences for competitiveness. What I mean by that is that women are just less likely to go for it; to go for the pay raise or go for the promotion. So that's what experimentalists find. So now you can ask yourself, "What causes women to be less competitively inclined than men?" So here, what we ended up doing is we go to the end of the earth, we find a society where women actually rule the society rather than men.

Tkac: Ohhhh (laughs).

List: There you go (laughs). It's the Khasi tribe near Shillong, India. So we go and run field experiments in India with these Khasi men and women, and the Khasis are a really interesting group. As you are riding out to the village you see signs, and you ask the driver, "What do those signs say?" The driver says, "Well, it's those male groups again, they're arguing for equality." And it's almost like if you have seen the Seinfeld episode, "The Bizarro World"...

Tkac: Exactly. No, that's right, it is. That's right. We've gone through the worm hole.

List: So it's the opposite. So this is what we have here. You knock on the door of a Khasi household and the man answers the door, and he brings you in to the back room and says, "You can talk to the woman of the household." And then the man goes and sits in the corner and he's quiet. So you have this opposite world, and we run experiments with these people. And we explore, are Khasi women like American women? Are Khasi men like American men? And what you find is something really interesting. A Khasi woman is a lot like an American man, and a Khasi man is a lot like an American woman.

Tkac: In terms of this competitiveness.

List: In terms of competitiveness, in terms of risk posture, you know, a willingness to take a risk, in terms of bargaining. So these households will send women out to do the bargaining in the market rather than the man. Exact opposite of what we have here.

So in this world you have women who are aggressive and competitive, and that leads us to believe that if you look at genetic differences between men and women—brain differences—there are some but they're very small differences in nature, but you have these very large differences in markets.

What those results suggest to us is that you can have small genetic differences and then they interact with the social environment. So the environment in the U.S. was (when I was growing up) was, if you're in gym and you're not trying as hard as you can as a boy, the gym teacher says, "Stop playing like a girl." Right? So you're told that you need to be competitive. And if you're a girl competing hard you are told, "Well, you're a tomboy" if you try too hard, or you're an even another name if you ask for things that aren't ladylike. Now what we find is that those social influences are extremely important in determining who we are and how we behave.

Tkac: I guess I'm just sort of curious, I'm sure that everything you have done has been through an IRB (internal review board), but in terms of thinking about experiments do you see effectively limits, or is the world so rich in terms of these kinds of field experiments that the kinds of things you might be concerned about, we can learn an awful lot without having to get anywhere close to what might be an ethical bound?

List: Yeah, I think we're not to the point yet where we are bumping up against constraints, in part because field experiments are so new.

You don't want to harm people. You always want to go through ethics boards and make sure what you're doing is right and just. But also there are certain questions that are really hard to answer when the person knows they're taking part in an experiment. In those cases, I think it's OK to run a natural field experiment, as long as a person is better off from being a part of your field experiment than if they weren't part of your field experiment.

Tkac: Well, it gets back to your earlier point when I asked about control versus the ability to extrapolate, and so what you're showing us is how people behave in their natural habitat, and then attempting to put enough structure so that we can understand why they're behaving a certain way.

List: If you want to know about the real world, you have to go and do experiments in the real world. And that's my research agenda.

Tkac: Great. Well, thank you very much, we've really enjoyed having you with us today.

List: Thanks for having me, that was a great interview.

Tkac: Thanks.