Samer Abdelnour posted "If Stoves Could Kill" on the Stanford Social Innovation Review blog the other day. Several people mailed me the link and asked me what I thought. Waving the red cape in my face, I'm afraid. And I have been to a bullfight, so I know it is not a good strategy to be a bull.
I wasn't bothered by Mr. Abdelnour's main point, about whether cook stoves really reduce violence against women in refugee camps. I think he makes some good arguments about that. But then he keeps going, and says "Research is beginning to question the ability for efficient stoves to effectively reduce fuel consumption and health risks." I think this statement, and the "research" it is based on, is misleading, erroneous, and dangerous.
To see the back and forth, look at the comments to the post. Heck, join in if you like what either of us has to say. I'll probably need help, as I am taking on the much exalted "Randomized Controlled Trial," and some people who are very smart, very well-intentioned (and much of whose work I admire*).
The problem isn't "If Stoves Could Kill"... instead, it is that their absence does kill. The use of traditional cooking methods and fuels kills many people every year. More than malaria. More than the tsunami. And some of this is preventable. With better stoves, better business models, more work. I don't know when, how or if the market will take off, and real impacts on health will be achieved. I think it will take time. And as the authors of the "Up In Smoke" paper say, it will take changes in behavior. So probably a long time. But I think it is more likely to happen from people that make stoves and sell them to other people, than from people that write reports.**
And what if, instead of asking about stoves, we turn the question to RCTs? What is the potential impact of misusing data, or poorly designed studies? One JPAL associated professor was quoted as saying "it is more satisfying to answer small questions well, than big questions badly." I would add that it is also dangerous to use answers to small questions to do anything other than answer small questions. A more accurate conclusion for the study might have been: "people in Orissa don't like using crappy stoves that don't work." Not, as Mr. Abdelnour states, to call into "question the ability for efficient stoves to effectively reduce fuel consumption and health risks."
Why do people seek to extend these studies' conclusions well beyond their boundaries? It seems to me that is a guaranteed way to answer bigger questions badly. If bad studies end good programs, the results really could be deadly.
__________
*here is a TED video of Esther Duflo, discussing vaccination, bed nets and education (but not cookstoves)
** Phil Auerswald put it well: "What works in development...is entrepreneurial exploration. Why? Because we don't know what works."
Thursday, September 20, 2012
If RCTs Could Kill
Subscribe to:
Post Comments (Atom)
4 comments:
Well said, Paul!
I am in agreement.
Paul
I think you're committing the same mistake that Abdelnour did--making too much of a single case. In his case, he uses the Orissa study to make a broad claim about cookstoves. Here you are using one person's expansive interpretation of that research to make a broad claim about RCTs and their proponents.
In the Duflo et. al. paper, the research is largely represented in the same way that you did: in a test of a specific stove in a specific place, the positive effects in the lab don't happen in the field. Their conclusion isn't a sweeping one about cookstoves but about the importance of testing technologies in the field not just in labs. I'm sure that's a sentiment you agree with just based on how you conduct your work.
Finally, I have to take issue with your representation of the findings as obvious and unsurprising. The program evaluated was conducted by an award winning NGO, Gram Vikas, which independently decided to deploy cookstoves and independently decided to use their particular stoves. It may have been well-known to you that these were crappy stoves but apparently not to well-regarded NGOs conducting such work. Similarly, the study cites work by the World Bank that indicates that this type and class of stove is the most commonly used with 166 million of them distributed.
While that by no means suggests that stove programs are useless, all stoves are ineffective or any other sweeping conclusion, it is good reason to look at any stove program to determine how well the supposed positive effects have been demonstrated in real world use.
That conclusion from the study is certainly externally valid and clearly useful.
Tim
Tim-
Fair enough... in this post I am reacting to a single instance. However, if you peel back to the other posts on SSIR and my blog that I have linked to, you will see I have concerns about other RCTs beyond "Up in Smoke".
You know more than I about this field- how broadly do you think RCT findings can be applied? Do you think that my concerns about borrowing this technique from medicine and applying it to social sciences have any merit? Dr. Duflo conveniently uses leeches as her way of jumping across this abyss.
And, yes, perhaps I know more about stoves... but since JPAL and others speak broadly about what works, it would have been great to have them use a stove that works. While this program in India accomplished a lot of installations, it was plagued by problems.
To be fair, better technologies were just coming into the market at the time the study was being done. I can't say "the researchers should have used technology" that either didn't exist, or was not well known at the time. However, since it is a working paper, I think some acknowledgement of these advances would be helpful to prevent over generalization of these findings.
I think some of my frustration is that I know of groups, including Envirofit, who have tried to get larger impact studies funded. As my first response to Abdelnour indicates, stoves are not a panacea. We don't know enough about the effects of reducing indoor air pollution... is it a linear or step function for health? How long does it take? We know that clean stove technology can reduce indoor air pollution by 80+%... but we don't have good causation data on improved health. Is that enough to reduce miscarriage rates? Eye disease? COPD? Is it akin to quitting smoking, or cutting back from 4 packs a day to 1 pack a day? We are working to get these answers, but they take time (http://www.cvmbs.colostate.edu/erhs/faculty/peel/j_peel.htm) Again, I think RCT could be useful on issues of effects of IAP because we are looking at human respiratory systems, not different cultural issues of adoption and behavior.
I appreciate your taking time to engage on this issue. If you were to advise the Global Alliance on Clean Cookstoves on how to take the Duflo et al study into account in future work, what would your advice be? If they were to fund an RCT, how would it be best designed to come up with more generalizable answers that could guide the targeting of cook stove programs? Paul
Here is a summary of a recent study by Prof. Peel on seeing some improvements in health in women using improved stoves. As she notes, however, many households did not completely adopt the new stoves (some kept their fires as well) http://medicalxpress.com/news/2012-11-cleaner-cookstoves-health-world.html#nwlt
Post a Comment