Provocateur-economist D^2 and the near-indescribable Chris Lightfoot together make a comprehensive defence of the Lancet Iraq 100,000 deaths survey.
Unless you’ve got a *very* strong background in statistical analysis, refusing to believe the results of this study makes you a hack – pure and simple. Even the pro-war Economist agrees the numbers are valid…
Are you kidding? The sampling for this was horrible, and the study’s conclusion was a range of something like 19,000 to 194,000, 100,000 was just a number somewhere close to the middle. Basically you and the press are saying that the forecast calls for rain sometime between Monday morning and Friday night, and that means it will definitively be pouring on Wednesday noon. Rubbish doesn’t come close to summing up this "survey".
I deduce from that comment that you don’t have a background in statistical analysis. Please let me know if I’m wrong.
Did you even read the Economist article. For those with tired mouse-clicking fingers, it points out that:
It does not, however, mean, as some commentators have argued in response to this study, that figures of 8,000 or 194,000 are as likely as one of 98,000. Quite the contrary. The farther one goes from 98,000, the less likely the figure is.
I haven’t done statistical analysis since business school nearly a decade ago, but if a report appears in The Lancet and its methodology is backed up by The Economist (which, as John rightly points out, would have had far more professional and ideological interest in debunking the findings than otherwise), it should be taken rather more seriously than just making unsupported cries of "rubbish".
Or rather, Timbeaux is at the very least going to have to explain in some detail why two heavyweight publications have taken such a gamble with their reputations, if their findings are indeed as off-beam as he implies. I’m also somewhat surprised that he hasn’t seen fit to append comments to either of the blog threads that John highlighted, given his breezy confidence that they’re barking up the wrong tree, so I’ll tactfully assume he’s composing some at this very moment.
Well, your wrong. They "are 95% confident that the war-caused deaths totaled some number between 8,000 and 194,000." Yeah? I’m about 95% confident that my checking balance is between $5 and $5 million…..seems I’m guaranteed to have at least $2,500,000…..sounds so great, I might even ask around, conduct a "survey" if you will.
http://slate.msn.com/id/2108887/
Jeez, if Slate even calls this bullshit, I’m tempted to think it might be case of someone looking for evidience of a pre-determined solution….
To even think that a questionaire survey is a remotely professional manner in which to determine hard statistics seems to put the onus of having a background in statistical analysis squarely on those who would believe such headlines on faith.
Extrapolating survey results….wish I could have pulled that one off in Engineering school….
As I said, feel free to append comments to the two cited blog threads that analyse this data in much more detail. Since you’re so confident that the Lancet</I> study is utterly bogus, you should have no difficulty running rings around their arguments.
I can only assume that you’re refraining from doing so because you’re worried about hurting their feelings. That shows a commendably tactful spirit, but I think it’s somewhat misplaced.
I just did run rings around their arguments, sorry you missed it. Let’s refresh: Extrapolation of data gathered by a questionnaire in the middle of a war? No medical records? No undertaker receipts? This is supposed to be scientific? They are "95 percent confident that the risk of death now is between 1.1 and 2.3 times higher than it was before the invasion"…between 10 and 230 percent? This is supposed to mean something?
This "study" doesn’t predict a bell curve as much as pancake. Epidemiology is useful for predicting recurring events, like cancer rates or suicides per, but let’s be serious here about the limits of statistical tools. Iraq Body Count puts it at 16,000, which means the actual is probably between 20 and 30k. Sounds to me like a war was a vast improvement over Saddam’s record.
Extrapolation is the alchemy of statistics, and it seems the Johns Hopkins team is making plenty of gold….have any of you ever read Darrell Huff?
How on earth can you claim to be "running rings around their arguments" when you posted a link to Fred Kaplan’s Slate piece as though it was the last word on the matter, despite this being after John gave us the link to Dsquared’s lengthy point-by-point demolition of its arguments?
Far from "running rings around their arguments", you’re clearly not even bothering to read the arguments in the first place, and merely restating the original objections to the Lancet piece, which both Dsquared and Lightfoot are explicitly debunking in considerable detail.
The only way you can convince me that you’re running rings around their arguments (or that you’re even capable of doing so) would be for you to go to their comments boxes and argue directly with them – and end up running rings around them too, of course, but that’s a given. Since you imply this should be a piece of cake, what’s stopping you?
I think everything that needs saying was said in the original post, and this comments tread demonstrates the truth of the second paragraph!
The household questionaire plus cluster survey approach is one which is quite widely used (mostly in investigating epidemics, but also in estimating casualties in wars) and works pretty well; see the references in the Lancet paper. If as you suggest the method were rubbish, why would none of the Lancet’s reviewers have pointed this out? You’re quite right that it’s not the ideal way to gather these kinds of statistics, but unfortunately it’s almost all that can be done in Iraq, given that the situation there is chaotic and dangerous, and the previous government wasn’t very hot on record-keeping.
Contrary to your claim, (and as you would discover from reading the paper) the investigators did ask for death certificates to verify at least two reported deaths for each cluster sampled (in practice they asked slightly more often than this); in 80% of cases where they asked, a death certificate was indeed shown.
A risk factor of 1.1 to 2.3 corresponds to "between 110 and 230 percent", not "between 10 and 230 percent", as you say.
Others have already pointed out the errors in your reasoning about the reported confidence interval. I suggest that you re-read the Economist’s piece and read the Lancet article itself before you try to develop your argument any further.
Iraq Body Count puts it at 16,000, which means the actual is probably between 20 and 30k
You do realise that when you say that, it’s an extrapolation, don’t you?
I’m about 95% confident that my checking balance is between $5 and $5 million…..seems I’m guaranteed to have at least $2,500,000
If your bank account really varied between $5 and $5m, then $2.5m would indeed be quite a good estimate. However, I would suggest to you that $5m is, actually outside the 95% confidence band for your bank account. If you are a normal American, then the 95% CI for your bank account would be more like $0-$100k.
However, Timbeaux, I would like to thank you for pointing out one mistake in my article. In the first footnote, I claim that "nobody has ever made a serious practical mistake [by taking an interpretation of the confidence interval as the interval in which you can say with 95% probability the true value lies]". You have just made a serious practical mistake of exactly that kind. May I link to your comment here in my correction?
Bottom lineis they are using predictive (i.e. future) tools to determine past events. That, and extrapolation is the worst form of prediction subject to the most error. That, and they don’t have the slightest confidence in their pre-war numbers, much less their post-war numbers. This kind of epi study would get you thrown out of a medical review board to howls of laughter, yet somehow it’s supposed to be definitive.
I think that the most tragic thing here is that Johns Hopkins has decided that politics should trump science, and no one seems to mind. People love being told what they want to hear.
People love being told what they want to hear.
And as a corollary, people hate being told what they don’t want to hear.
I presume you think The Lancet is somehow different from a "medical review board". Maybe I was just imagining it to be one of the world’s leading medical journals…
I just think that using a tool useful for predicting random or pseudo-random events, such as cancer, is not in the least useful for predicting hard numbers of non-random events. Couple that with a very low confidence in estimating excess deaths from a baseline on which there is very low confidence…..the reported number is meaningless. I’m quite certain that the Hopkins team ran their numbers proper for the chosen methodology, it’s the methodology they chose that stinks to high heaven.
Is 98,000 the most likely result from their numbers? Sure, but what’s the relative meaning of that? Is a 2% level of confidence something to run a headline on? The R^2 on this study has to be the size of a small city, from what I can tell (hard to tell for sure, the Lancet is having "technical difficulties"), and they can’t even be confident of that since they have no decent confidence in their baseline. It’s shooting a moving target from a moving platform, while blindfolded, and declaring you’re 95% likely to have hit a target a mile wide. The number say so, it must be "the most likely" outcome.
Would I be suprised or even slightly alarmed to hear that more Iraqis died during the year after the war than the year before? Of course not. But to publish such a study, using a headline for which the study gives very low confidence, is pure politics. The BBC can engage in it, CBS, Nature, they’ve all been called on politically motivated BS. Johns Hopkins almost lost their accreditation over ethics violations a few years back. What makes them and the Lancet above tough questions they can’t answer?
"… The R^2 on this study has to be the size of a small city… It’s shooting a moving target from a moving platform, while blindfolded… The number[sic.] say so…."
Fine words. However, they make it plain that you (a) don’t know what you’re talking about; and (b) have no substantive complaint about the study.
Since you seem to be having trouble downloading the paper from the Lancet site, I’ve put a copy here. Perhaps once you’ve read it you’ll find a killer objection? I’m not holding my breath.
Chris, you really don’t see the problem in counting excess deaths without a firm baseline? Using extrapolative epi techniques for non-random, hetrogeneous events? Publishing a mean value for which there is very little probability confidence? A 95% confidence spread over a 186,000 life range?
Oh, and I’ve been struck by something else. How in the world can this study claim that medical records are an unreliable indicator, yet at the same time try to confirm their own results by saying that 81% of (certain) respondents could back it up with documentary evidence? It can’t be had both ways. Did they only ask respondents close to hospitals if they had death certificates? Either way it doesn’t add up.
Look, I made my peace with the war a long time ago; I’m not trying to defend Bush or Blair or Middle East domino theories or DU munitions or anything else (although I would if they were the subject at hand). I do have a big issue with prestigious journals publishing questionable studies purely for headline value. It seems to be more the norm these days then at any point that I can remember. I also take issue with being told that on reading a study of such a devisive political nature, that questioning the motives or honesty of either the surveyors or the surveyed is the definition of "hack", as Daniel would have me believe. The onus is not on me to disprove their work, the onus is on them to present a convincing conclusion against all reasonable questions. This study does not do that IMHO.
Chris is being polite about this, but I won’t; your comment about the R^2 reveals that you have no clue what you’re talking about; you just heard that this study was about statistics, and you remembered from some way back that the R^2 ratio had something to do with statistics, so you thought you’d drop it into the conversation in order to look like you knew a lot about the subject.
This analysis was not a regression, therefore there was no regression sum of squares and no residual sum of squares, therefore talk about R^2 is meaningless. In any case, a large R^2 is generally a good thing, not a bad one. You are also using the phrases "non-random" and "heterogeneous" in the same way.
You’ve done the statistical debate equivalent of soiling yourself in public now; you really can’t expect to be taken seriously on this subject again.
If you’ll read my article (unlikely; you didn’t read the Lancet article or Chris’s, so why should I be special), you’ll see that my definition of "hack" revolves round people questioning the honesty of the researchers _without_having_any_basis_for_doing_so_ . If the cap fits, and it does, wear it.
I wll also add six "hack points", on a scale to be determined later, for claiming to have "run rings around" arguments that you didn’t understand at all.
I will respond to your other points as and when (and only when) you can find someone else to make them; someone who has not embarrassed themselves in my eyes by pretending to talk knowledgably about the R^2 ratio for something that was not a regression. I apologise if this appears high-handed, but I simply cannot allow this kind of behaviour to go unremarked.
R^2, standard deviation, whatever. My point there was the curve is too flat to make any kind of statement like the Lancet did. It was done purely to attract headlines, which is rather sad. My broader point is that they are using tools unsuitable to the task, which is not to say that there ARE any suitable tools, but passing this off as something meaningful is rather absurd. By the bye, they are still having "technical difficulties", seems like they may have pulled the damn thing in shame.
"R^2, standard deviation, whatever." Oh god, this is excrutiating. Timbeaux: stop. Just stop.
What’s excruciating is being condescended to by D^2 about my use of terms, when I think the conceptual questions I raised were put forth rather clearly, and he them avoided entirely (with his nose rather high in the air).
This study did not include even a single cluster from Basrah province, containing Iraq’s second-largest city, and which has been the calmest area of the entire country. Yet "There is no realistic way in which a critique of this sort [bias] can get off the ground". Supposedly these clusters were chosen randomly, yet the largest, calmest region in the nation has mysteriously been bypassed entirely, in favor of a generous distribution across the Sunni Triangle. Hmmm….ya think that might scue the results of an extrapolation slightly? Error magnification, anyone?
Ultimately, as I said before, I could care less. The headline value of this "study" tanked, once most reasonable people realized that the headline was meaningless, and what should really have been taken from it is that violence is the leading cause of death in Iraq. Wow, my seven-year old could have told you that, and much cheaper to boot.
Basrah governorate (Iraq is divided into governorates, not provinces) was grouped together with Missan governorate, which has, if anything, been even calmer.
The claim that peaceful areas have been intentionally taken out of the sample is ridiculous; Najaf governorate was grouped with Karbala and Qadisayah was grouped with Dhi Quar. This meant that two of the most violent cities in Iraq (Najaf and Samarra) had no sampling at all, and the governorate containing Samarra was grouped with a largely Shi’ite province.
The conceptual questions you have raised are indeed clear, in as much as it is clear that you are talking rubbish. I still refuse to discuss specifics with you until you apologise for trying to bullshit me about the R^2 ratio, however. Since you have apparently admitted the survey’s main conclusion (you have admitted that the leading cause of death in Iraq is violence; since violence was not the leaeding cause of death in Iraq before the invasion, you must therefore admit that the death rate has gone up), however, this discussion would seem rather pointless, unless you are deriving masochistic pleasure from it.
It is appropriate to raise one’s nose in the air, by the way, when someone you are talking to has a visible and spreading brown stain on his trousers.
It is appropriate to raise one’s nose in the air, by the way, when someone you are talking to has a visible and spreading brown stain on his trousers.
Does that go for your post on Clark County, Ohio, too?
I find the Lancet study laughable on several grounds, quite apart from how valid it juggles numbers.
1) The conclusion. A 95% confidence interval of 8000-194000? Are you joking? What does that tell us that’s useful? Nothing.
2) The prewar estimates. Contradicted by UN estimates.
3) The authors. They make no pretense of objectivity. They only agreed to do the study if it could be publshed before the elections, so it could help defeat Bush.
4) Common sense. If 185 more people died every day, where are the headlines? We get a bulletin every time an Iraqi stubs his toe, along with 20 editorials claiming it proves the war was wrong. It’s not reasonable to suggest hundreds of massacres have gone unreported.
I could go on.
Go read the Crooked Timber post. Then fuck off.
Matt, I’d like to respond to what you pointed out from the Economist…
It does not, however, mean, as some commentators have argued in response to this study, that figures of 8,000 or 194,000 are as likely as one of 98,000. Quite the contrary. The farther one goes from 98,000, the less likely the figure is.
I would like to ask you this, as a novice statics person.
Couldn’t one also say that the closer one gets to 98,000 the lower the confidence you have. Everyone uses 95%, but what if the calculation was for 50%. I didn’t take the time to look up the sample size, or the standard deviation or standard error, but I do know that as you lower the confidence you tighten the band.
as a non statistician I actually prefer Timbeaux reasonements above the snobbish parlay of The Lancet defenders(dsquared,lightfoot). pointing to pants stains does not look like winning the argument does it?
(frustrated?) Big-mouths shouting behind the backs of big money publicists like Teh Lancet’s, I dl call them. stick with the establishment guys, it can never go wrong..ofcourse the earth is in the middle of the universe..
it defies me how "clustering" can bring more accurate information compared to a geographically completely spread sample. to me this should lead to loss of accurracy. that this is not the case ,seems one of the big points in the Lancets small sample sized study. they refer to previous done bio statistics where it seems to work well. but how can bio statistics be compared or related to political/social sensi, and to the case of victimisation in families ? to go and ask anxious families if and how family members died, during instable times, is a political sensus. Remember opinion polling is something Iraqis have not seen in 3 generations.
the snobs dont get it in this above discussion, I think.