Roy Head: How to put your own charity to the test

Mass media can reach millions of people, but can it improve health and save lives as effectively as other top interventions? Past studies of mass-media campaigns for public health failed to find evidence of strong impact, belying the potential of the medium. In this talk, Roy Head, CEO of Development Media International (DMI), discusses the randomized controlled trial DMI ran on its own health campaign — and the surprising results.

We’ve lightly edited Roy’s talk for clarity. You can also watch it on YouTube and discuss it on the EA Forum.

The Talk

Good afternoon.

Last night, for the first and what will probably be the only time in my life, I attended the celebration ceremony for the reception of the 2019 Nobel Prize for Economics given to Michael Kremer, Abhijit Banerjee, and Esther Duflo. I was there because I'd worked quite a lot over the past few years with [Michael Kremer’s] wife, Rachel Glennerster, who is one of the founders of the Poverty Action Lab at MIT.

I will say that if you have any anxieties about underachievement, finding out that the date of birth of the person who received the Nobel Prize is within one month of your own is pretty savory. If I’d been born just one year earlier, there'd still be time. [Laughs.]

Anyway, [Michael Kremer] was thanking his wife — which you would expect because they’ve worked very closely together — as well as everybody in the room. And these weren't politicians’ thank yous (e.g., “This wasn't a victory for me — this is a victory for you all”). They weren’t false modesty. I suddenly realized that everybody in the room had been involved in some way in the work of doing RCTs [randomized controlled trials], analyzing them, or allocating money based on them. Everybody in that room was committed to the principle of evaluating what works in a very rigorous way for the benefit of some of the poorest people in the world. And suddenly, I felt very proud.

I hadn’t expected to feel proud at all. And I just want to convey that to [those of you in the effective altruism movement] as well, because I think that everybody in this room, to some small extent, shares in that Nobel Prize. Michael Kremer was really giving that Nobel Prize to the people who have created not just a few academic papers, but a movement. So I think it's something for all of us to share in.

Having said all of that, and having approved of a randomized controlled trial [to gauge Development Media International’s work], justifying your existence is a very, very sobering experience. It’s not based on whether you honestly intend to generate good results, whether you're doing good work, whether you're efficient, or whether you have a nice team. Your value is assessed simply in terms of a few specific numbers.

I want to share what that experience was like for Development Media International (DMI), the organization that I run. Let me start with a few numbers underpinning what we do.

This is a list of the major causes of death in 2016. Cardiovascular disease is at the top, and natural disasters are at the bottom. If we were omnipotent — if we could do anything we wanted — what would we do?

Maybe we would try to stop all wars. Maybe we'd try to stop all murders, all terrorism. Well, let's look at that. Stopping all the murders in the world would save 390,000 lives, stopping all the wars in the world would save 115,000 lives, and stopping all terrorism would save another 34,000 lives in addition to that. That’s over a half-million deaths averted, which sounds like a lot, even though it's completely unachievable.

In comparison, all of the malaria deaths in the world alone amount to over 700,000, all of the diarrhea deaths total over 1.6 million, and all the pneumonia deaths and lower respiratory infections total almost 2.4 million. And these are problems we can do something about.

That's why DMI exists. It aims to use mass media to do something about [problems we can effectively address]. We focus very tightly on those three issues: malaria, diarrhea, and pneumonia, which are three of the biggest causes of death among children under five years old. We run very large mass-media campaigns. We broadcast, on average, 10 times a day, 365 days a year, via radio, television, and mobile phone. We work in 10 countries, and we focus mainly on child survival, but we also do some early child development work, and some family planning work.

But does it work?

There’s a lot of before-and-after evidence showing that if you run a media campaign, you’ll see better results at the end of the campaign. But there are weaknesses with before-and-after data. Rigorous randomized controlled data for the mass media sector is almost nonexistent. There have been four randomized controlled trials done. They were all done in the United States, they all failed, and the problem was insufficient exposure.

You can easily do a campaign in Chicago, LA, Cleveland, and New York — and not do it in Miami and five, six, or 10 other cities. That's easy. But to do that, you must leave out NBC, ABC, CBS, Disney, and all of the national radio and television stations. So you're doing a mass-media campaign [without] the really big value of mass media, which is reaching millions, or tens of millions, of people at a time. You're doing it with two hands tied behind your back, because you don’t have those mass audiences.

So, [measuring that type of limited campaign] isn’t a real trial of what mass media actually does. The RCT method gets in the way. It's never been attempted in a developing country, and as a result, mass media has been confined to the periphery of public health: "I'm doing a tuberculosis campaign, maybe we should do some mass media on the edge," or "Maybe we should do some extra media stuff." Mass media is not seen as a core component of public health.

We tried to remedy this in two ways. First of all, we developed a saturation theory that we published in the Lancet. It's about broadcasting 10 times a day, which sounds extremely crude — and it is. It focuses on just the principle of changing behaviors; it's not about doing a finely crafted, spotlit broadcast once or twice. You have to hit people with [your message] 10 times a day. If you hit the audience 10 times a day, you'll probably reach any given individual two or three times a day.

We also discovered something unique in Burkina Faso, which is a country in West Africa between Ghana and Niger: The national media reaches only about 5% of the population. The country made a political decision to broadcast almost exclusively in French, and in Burkina Faso, people speak all sorts of languages and prefer to listen to their own local radio stations, in their own local languages. That means if you broadcast in certain zones — represented by the blue zones in the slide above, as opposed to the red zones, where we didn’t broadcast — the local FM radio stations capture pretty much all of the audience.

We conducted a randomized controlled trial designed to reach nearly all of the people who listen to the mass media. We did it broadcasting in seven areas, with seven control areas. We were broadcasting for 35 months, 365 days a year. We aired 10 spots a day, plus two hours every night, in six languages. We focused on malaria, diarrhea, and pneumonia.

Let me just give you an anecdotal feel for how the campaign worked.

[Roy plays a campaign video. A resident of Burkina Faso describes, in his native tongue, how his daughter contracted malaria. He didn’t understand what had happened and went to local healers, to no avail. Then, a neighbor told him about a radio message describing the symptoms of the disease and urging parents to take children who exhibited them to the health center. The narrator did so. His child was diagnosed with severe malaria and, with proper treatment, recovered in one week. The narrator purchased a radio in order to have access to similar messages.]

This film captures that there's nothing more important than a child not dying. At the same time, we needed to know that our radio messages affect more than one child, because we wanted to know whether this is an efficient way of saving children's lives. The film is illustrative, but it's absolutely not enough. We had to do a survey.

When we first received the survey results, we had an enormous problem.

We couldn't detect a fall in mortality. And that was because the survey had an 80% power to detect a 20% fall in mortality. But there was no way of powering any study in the world better than this. With unlimited funds we could not have devised a design that would detect mortality reductions of less than 15%. It was impossible. There's no country in the world where you could do that.

It was also difficult to manage this project. We were broadcasting 70 hours a week of radio, plus 10 spots a day, working in six languages, and trying to cope with all sorts of interventions in our control areas. That was a managerial challenge. But we got through that part. Then came this challenge with the survey, which was pretty sobering.

Thankfully, we were saved by our second line of defense. The other thing that we'd nominated at the beginning of the trial as a major result was administrative data from the health centers themselves.

This is the data that came out of the health centers in Burkina Faso. It’s from 600,000 consultations. The scale is the number of consultations per month. And there are a lot. It went up to 15,000 consultations in one month. The solid line represents the number of consultations in the intervention clusters, and the dotted line the number of consultations in the control clusters. They're pretty much together before the campaign starts, and then there’s a separation as the campaign starts — and a widening gap, certainly in the first year, and also in years two and three.

When you do a time-series analysis on that data, and compare the ratio of those two lines, the result is a 56% increase in malaria diagnosis and treatment in year one, a 37% increase in year two, and a 35% increase in year three. And because this data follows the malaria seasons, it looks like good data. The lines follow each other, and when you do those statistical tests, the results are robust.

For pneumonia, there was a 39% increase, and for diarrhea diagnosis, a 73% increase that, interestingly, went up to a 107% increase in year three.

This was great. It was the first time ever that mass media had been shown to change behaviors. There was previously no data in the entire advertising world, or the entire epidemiological world, doing that.

One other piece of data I’ll mention is that there was no change in [visits to health centers for] upper respiratory infections. We didn't campaign on upper respiratory infections, which are coughs and colds. They don't matter; they aren't going to kill kids. And it was very reassuring when the data showed that we hadn't just driven everybody to the health centers no matter what they'd contracted — that when they came to the health center, they had come because of symptoms related to the diseases that we campaigned for, which were malaria, diarrhea, and pneumonia.

To determine whether the campaign saved lives or not, we used the Lives Saved Tool. It's the gold standard in epidemiology. For example, if an organization has increased breastfeeding by 30% in Angola, they can input that data into the tool and determine how many lives they’ve saved. We used the figures that we had for malaria, diarrhea, and pneumonia — plus there were some other figures — and derived the number of lives that had been saved.

During the trial period, we reckoned that we had saved 3,000 lives. Then, when we scaled up the campaign in Burkina Faso and in Mozambique, we determined we would save 10,000 lives in Burkina Faso and 21,000 lives in Mozambique. And the cost per DALY (disability-adjusted life years) ranges from $7 to $27.

That's significant, because we're broadcasting information into thin air. This is radio programming. This isn't a hospital. There's not even staff. You can't really see [our work], and hence, people are somewhat skeptical about it. We have more to prove than most organizations. Malaria bednets are proven to work. And even if there are no RCTs for caesarean sections, which we wouldn’t ever do an experiment on, you can see that the mother and child live. It’s very, very clear. But is mass media worth spending money on — money that, obviously, could be spent on other things?

Because we were able to calculate the cost per DALY, we were then able to compare it to other child health interventions. This table is an excerpt from the Disease Control Priorities Project (DCP), which is the Bible of health economics. They compare the evidence for different child health subjects — in this case, child health interventions — and they use a logarithmic scale. It goes from $1 to $10 to $100 to $1,000 to $10,000. And the costs range from the very cheapest thing, which is artesunate for malaria, up to gender training for intimate partner violence. It ranges from $7 to $2,300. [Our mass media intervention comes] in near the bottom [in terms of cost].

It's very difficult to say what our exact rank is. This scale really shows orders of magnitude. When you're comparing RCTs from different countries, you can't say definitively, "This is number two, this is number three, this is number four." But what you can say is that mass media is at the very cheapest end of our public health interventions for child health.

This is reasonably satisfying. We're very pleased. In a way, it confirms what we suspected: that mass media is just a vehicle for human knowledge, and human knowledge determines when to see a doctor. And we try to put human knowledge, and mass media as a vehicle for advancing human knowledge, right at the center of public health.

[These results] didn't cover the cost of doing the RCT. It took us five years to do the trial. It took us three years to raise the money for the trial. It took a few years to analyze it afterwards. It was extremely expensive. I've got gray hairs as a result. It was an ordeal, and we've done some other trials since then, which have been rather easier to run, but that was our experience with this particular trial, and I wanted to share it.

Thank you.

Moderator: I really appreciated that your presentation had a lot of details in it. But there are some bigger picture things that I'm wondering about as well. Can you talk a little bit about whether you have any principles that guide your approach to staffing, and is that different here [in the United Kingdom] versus in the locations where you provide the intervention?

Roy: Yeah. Staffing is huge. An organization is simply a bunch of ideas and a bunch of staff. That's all it is. And so, nothing that we do can be done without the people we recruit. And our criteria in London for any job is the person be smart, nice, and really want the job. It's as simple as that. Obviously, if we're recruiting epidemiologists, a PhD will be helpful, but it still boils down to smart, nice, and really wants the job.

In a developing country, it's very different. We don't use paper qualifications at all. It's very easy, when you're working somewhere like Burkina Faso, to simply steal staff from the other NGOs [non-governmental organizations]. It's this small group of elite people who have master's degrees and go from NGO to NGO.

It's not at all a good way to build the capacity of the country. When we were working in Burkina, we advertised. We went to town hall meetings, schools, and community centers. We said, "Look, we're interested in you. You don't need to have any qualifications at all. You do have to be able to write French, but your application form will not be your degrees. Your application form will simply be a script that you're going to write in French on this subject."

We received 600 applications, and we tested about 80 of them, and we recruited about 15 of them. And one of my favorite script writers had been a security guard. His father died when he was 11, and he had to find work. And yet, he was talented. He had no way to go to university and get those qualifications, but he was great. [Emphasis ours.] And so I believe in the approach we used as a way of trying to recruit people. We don’t just rely on the very small circle of people that most organizations use.

Moderator: That's fantastic. You did mention that there were some trials that you've done since the more challenging one — trials that have maybe given you fewer gray hairs. Are there any you’d like to provide some little snippets about or add?

Roy: Yeah. We've just finished a randomized controlled trial with Rachel Glennerster. It was on family planning, and she's excellent, and it was a much easier trial to do. We haven’t published the results yet, but they are clear-cut: We improved the modern contraceptive prevalence rate by 17%. We're happy with that, and I think it has major implications for how family-planning campaigns are conducted.

We are just about to start an RCT on early childhood development. The idea is you can teach parents to stimulate their children before they’re three years old, because if those neural connections aren’t made in their brains by then, they're not going to be made. So that's something that we hope improves the IQ of a generation, if you like.

Moderator: Great. That sounds like amazing work. Thank you so much for this presentation and for being here.

Roy: Thank you very much.