Regular readers know that I follow Nate Silver at FiveThirtyEight closely this time of year.
The blog is based at the New York Times, but Silver developed his own methodology over the years for analyzing available polling data for national elections (and other things too). The Times lured him under their umbrella after the 2008 election, but he has his own brand and could easily go back out on his own.
I hope he does, actually.
A few days ago, the NYT’s Public Editor Margaret Sullivan posted Under Attack, Nate Silver Picks the Wrong Defense. Sullivan seriously compromised her own credibility in that piece.
Short background: MSNBC’s Joe Scarborough had apparently been criticizing Silver’s work (and engaging in personal attacks), so via Twitter Silver bet him $1000 that Obama would win. Nate’s many things — a statistician, a bettor, someone who believes in his own work, a blogger. He’s not a traditional journalist and he’s not a Times reporter who is expected to maintain an appearance of neutrality. Silver’s model shows Obama with a better than 80 percent chance of re-election, so the $1000 bet is a pretty good one (although he could still lose — close to 1/5 of the time).
Sullivan took Silver to task for that bet, which could be fine given the Times’ standards but in practice is really problematic given the hazy definition of Silver’s position.
And it’s the way that Sullivan went about it that I object most strongly too.
She writes:
For months now, he has been predicting that President Obama has about a 75 percent probability, give or take a few points, of winning re-election on Tuesday. He uses an algorithm – some call it a secret sauce — that combines the numbers in public opinion polls and produces a result that he then turns into a prediction.
Since the summer, the model has shown Obama with a lead, but his actual chances of winning on Nov. 6 have varied from 59 percent to 89 percent. That’s hardly “a few points” — and that’s precisely the sort of lazy characterization all too common in news coverage. And that’s precisely the sort of lazy use of numbers that Silver himself would never allow.
And why echo the derisive “secret sauce” complaint? Some mathematical details can’t be released lest Silver undermine his own value, but he’s been exhaustive — really — in explaining over the months what data goes in, how data cycles out, how various weights of information change over time, why some pollsters are weighed more heavily or are adjusted for their partisan “house effect”, and so forth and so on.
Again, Sullivan’s decision to use the derisive “secret sauce” term coupled with her own vague understanding of the model’s methodology insults both Silver and his readership.
And I should add that Silver is not really in the “prediction” racket. He’s determining odds of a particular outcome, not necessarily predicting one.
Sullivan writes:
It’s also inappropriate for a Times journalist, which is how Mr. Silver is seen by the public even though he’s not a regular staff member.
“I wouldn’t want to see it become newsroom practice,†said the associate managing editor for standards, Philip B. Corbett. He described Mr. Silver’s status as a blogger — something like a columnist — as a mitigating factor.
Granted, Mr. Silver isn’t covering the presidential race as a political reporter would.
But he is closely associated with The Times and its journalism – in fact, he’s probably (and please know that I use the p-word loosely) its most high-profile writer at this particular moment.
FiveThirtyEight’s electoral predictions three days before the election
But consider Sullivan’s formulation: it’s inappropriate for a Times journalist to make a bet on Twitter; therefore it’s inappropriate for a non-staff blogger who is not a reporter but who is a little like a columnist. In an earlier post, executive editor Jill Abramson was quoted as saying that Silver is “a separate entity, somewhat analogous to that of a columnist.”
Really, what the hell?
Sullivan backtracked a bit with a followup post, but she misses the point there too.
A few observations:
Journalists are scared of numbers
Not all of them certainly, but this is a huge problem in news coverage today. For example, if more reporters had understood simple concepts like months of inventory, we would have seen a much better public grasp of the hubris of the housing boom. What Silver is doing at FiveThirtyEight seems to scare people who prefer words over numbers. Silver’s a good writer, a clear writer, but he’s not going to win any awards for his long-winded prose. He uses words to give voice to the data he collects and analyzes. Bill McBride does the same thing at Calculated Risk. Such approaches are challenging to the irrational world of punditry that dominates so much of our public discourse.
The NYT and other legacy media are struggling to adjust to the world of new media
As I note above, the Times clearly needs to figure out exactly what Silver’s content is. If the NYT itself doesn’t know, then how can they possibly criticize him like this based on non-existent standards?
News coverage is biased toward horse races and “tossups”
Romney could still win on Tuesday, but it’s not a tossup. Check out Silver’s post about Friday’s polling: Nov. 2: For Romney to Win, State Polls Must Be Statistically Biased. Of 22 swing state polls (CO, FL, IA, ME-2, MI, MN, NH, OH, VA, WI) released on Friday, two were tied and Obama led in 19 of them. Romney led in one poll in Florida. And that’s not just a single day — Obama has maintained small but steady leads in more than enough states to get to 270 electoral votes. From that post:
If you have just one poll of a state, the statistical sampling error will be fairly high. For instance, a poll of 800 voters has a margin of error in estimating one candidate’s vote share of about plus or minus 3.5 percentage points. In a two-candidate race, however, the margin of error in estimating the difference between the candidates (as in: “Obama leads Romney by five pointsâ€) is roughly twice that, plus or minus seven percentage points, since a vote for one candidate is necessarily a vote against the other one.
The margin of error is much reduced, however, when you aggregate different polls together, since that creates a much larger sample size. In Ohio, for example, there have been 17,615 interviews of likely voters in polls conducted there within the past 10 days. That yields a margin of error, in measuring the difference between the candidates, of about 1.5 percentage point — smaller than Mr. Obama’s current lead in the polling average there.
Legacy media need to figure this thing out — and fast. For informed news consumers, it’s now almost laughable for media outlets to give focused attention to an individual poll, as major publications and stations do routinely for polls they have commissioned.