If RCTs Could Kill

Samer Abdelnour posted "If Stoves Could Kill" on the Stanford Social Innovation Review blog the other day. Several people mailed me the link and asked me what I thought. Waving the red cape in my face, I'm afraid. And I have been to a bullfight, so I know it is not a good strategy to be a bull.  

I wasn't bothered by Mr. Abdelnour's main point, about whether cook stoves really reduce violence against women in refugee camps. I think he makes some good arguments about that. But then he keeps going, and says "Research is beginning to question the ability for efficient stoves to effectively reduce fuel consumption and health risks." I think this statement, and the "research" it is based on, is misleading, erroneous, and dangerous. 

To see the back and forth, look at the comments to the post. Heck, join in if you like what either of us has to say. I'll probably need help, as I am taking on the much exalted "Randomized Controlled Trial," and some people who are very smart, very well-intentioned (and much of whose work I admire*).

The problem isn't "If Stoves Could Kill"... instead, it is that their absence does kill. The use of traditional cooking methods and fuels kills many people every year. More than malaria. More than the tsunami. And some of this is preventable. With better stoves, better business models, more work. I don't know when, how or if the market will take off, and real impacts on health will be achieved. I think it will take time. And as the authors of the "Up In Smoke" paper say, it will take changes in behavior. So probably a long time. But I think it is more likely to happen from people that make stoves and sell them to other people, than from people that write reports.** 

And what if, instead of asking about stoves, we turn the question to RCTs? What is the potential impact of misusing data, or poorly designed studies? One JPAL associated professor was quoted as saying "it is more satisfying to answer small questions well, than big questions badly." I would add that it is also dangerous to use answers to small questions to do anything other than answer small questions. A more accurate conclusion for the study might have been: "people in Orissa don't like using crappy stoves that don't work." Not, as Mr. Abdelnour states, to call into "question the ability for efficient stoves to effectively reduce fuel consumption and health risks." 

Why do people seek to extend these studies' conclusions well beyond their boundaries? It seems to me that is a guaranteed way to answer bigger questions badly. If bad studies end good programs, the results really could be deadly. 
__________
*here is a TED video of Esther Duflo, discussing vaccination, bed nets and education (but not cookstoves)
** Phil Auerswald put it well: "What works in development...is entrepreneurial exploration. Why? Because we don't know what works."