By Kevin Meyer
In this edition of our irregular Fun with Statistics series we take a look at an oft-unforeseen pitfall of analysis: inadvertently framing the analysis. How often do we ask our subservient underlings for some report or analysis, but then skew that analysis by drawing boundaries around potential outcomes? We may think we're just providing "guidance," but where does guidance end and prejudicial blinders begin?
A good article in The Atlantic points out this potential pitfall using one of our, uh, favorite fellows - Paul Krugman - as an example.
Paul Krugman has an interesting post up on how to read a CBO document. He highlights the fact that the CBO reports using the assumptions it is required to make, not any estimate of what is particularly likely to happen in the real world:
What you need to realize is that the CBO is the servant of members of Congress, which means that if a Congressman asks it to analyze a plan under certain assumptions, it will do just that -- no matter how unrealistic the assumptions may be. CBO will tell you what's going on, but it will do so deadpan, doing nothing in terms of emphasis or placement to highlight the funny business.
Now of course the interesting angle to this is that Krugman himself often used CBO numbers to justify some of the nonsense he was bloviating about, even just the prior day in an article on how the Medicare Trustees believes that changes resulting from healthcare reform will reduce the deficit. As Megan McArdle of The Atlantic continues:
I could wish that he had gone over this topic just a wee bit earlier--like yesterday, when he argued that the Medicare Trustees report shows that we've done a whole bunch of deficit reduction:
In other words, the Medicare actuaries believe that the cost-saving provisions in the Obama health reform will make a huge difference to the long-run budget outlook. Yes, it's just a projection, and debatable like all projections. And it's still not enough. But anyone who both claims to be worried about the long-run deficit and was opposed to health reform has some explaining to do. All the facts we have suggest that health reform was the biggest move toward fiscal responsibility in a long, long time.
There are two problems with this claim. The first, as I noted earlier this week when Health and Human Services Secretary Kathleen Sebelius made a similar claim, is that it requires double counting. If the money is used to extend the program's trust fund, then it can't be used elsewhere.
The second is that Medicare's actuaries aren't confident that the program cuts called for by the PPACA will actually happen. For example, page three of the report's introduction notes that the headline estimates assume that Medicare payment rates to physicians will be cut by a total of 30 percent over the next three years despite "virtual certainty" that legislators will override the scheduled reductions. Indeed, in a somewhat extraordinary caveat, the Trustees report explicitly warns readers not to presume that its top-line cost estimates represent the most likely outcome: "It is important to note that the actual future costs for Medicare are likely to exceed those shown by the current-law projections in this report."
I wish Paul Krugman would do some explaining--about the source of his differential skepticism.
In this case the Medicare Trustees and actuaries actually found a way to insert a bit of their own professional opinion, constrained within the artificial boundaries imposed on them by those that requested the report: that model of fiscal prudence themselves, Congress.
There's a good lesson for us: be careful of constraining analysis by "guidance" or preconceived opinion. You may get what you want, but it may be incorrect.