Analyzing the Lancet numbers

This is probably the best piece I’ve seen on those questionable Lancet numbers yet. Steven Moore checks out the methodology and finds it extremely faulty:

After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don’t get me wrong, there have been far too many deaths in Iraq by anyone’s measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5%–not 1200%.

The group–associated with the Johns Hopkins Bloomberg School of Public Health–employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in “clusters” within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.

However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, “Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey,” the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn’t survey a junior high school, no less an entire country, using only 47 cluster points.

Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.’s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711–almost three times that of the Johns Hopkins team for 93% of the sample size.

Here are Moore’s credentials, BTW:

Mr. Moore, a political consultant with Gorton Moore International, trained Iraqi researchers for the International Republican Institute from 2003 to 2004 and conducted survey research for the Coalition Forces from 2005 to 2006.

Make sure to read the whole thing.

Hat tip: ST reader Severian

Prior:

• About those Lancet numbers
• Report on 655,000 alleged Iraqi civilian casualties since the beginning of the Iraq war: the latest October surprise