Now there’s an excellent piece in the Huffington Post about that issue and several others that could have contributed to Gallup’s failings: Gallup Presidential Poll: How Did Brand-Name Firm Blow Election?.

From that piece:

Since the election, the Gallup Poll’s editor-in-chief, Frank Newport, has at times downplayed the significance of his firm’s shortcomings. At a panel in November, he characterized Gallup’s final pre-election poll as “in the range of where it ended up” and “within a point or two” of the final forecasts of other polls. But in late January, he announced that the company was conducting a “comprehensive review” of its polling methods.

There is a lot at stake in this review, which is being assisted by University of Michigan political scientist and highly respected survey methodologist Michael Traugott. Polling is a competitive business, and Gallup’s value as a brand is tied directly to the accuracy of its results.

The firm’s reputation had already taken a hit last summer when an investigation by The Huffington Post revealed that the way Gallup accounted for race led to an under-representation of non-whites in its samples and a consistent underestimation of Obama’s job approval rating, prompting the firm to make changes in its methodology. (Since Gallup implemented those changes in October, the “house effect” in its measurement of Obama’s job rating has significantly decreased.)

It’s a really interesting piece that discusses myriad issues that polling companies are facing in the 21st century.

Highly recommended for those who followed the numbers closely last year. Those numbers were pretty clear if one trusted the data. Ironically, Gallup didn’t trust its own data enough. If they had reported their registered voter polling versus their likely voter polling, they would have been considered one of the best performers of the election cycle.

Gallup will no longer be conducting polls for USA Today, the two organizations announced Friday.

Both said the split, after 20 years of collaboration, was a mutual decision based on the changing media and polling landscape.

“Given these shifts, Gallup and USA Today have made a mutual decision to move in independent directions beginning in 2013, and Gallup will evolve the polling it conducted in partnership with USA Today in some different and new strategic directions,” Gallup said. “As it has been, Gallup.com will remain the primary source for Gallup polls conducted in the U.S. and around the world.”

USA Today said in its own statement that it is already in negotiations with another pollster.

Well let’s hope that whatever pollster USA Today ends up partnering with in upcoming elections, the newspaper has at least learned some key lessons from 2012. It doesn’t look like Gallup has learned any.

I wrote a number of posts critical of Gallup both before and after the general election in November. In mid-October, reading the work of Nate Silver and others, I even asked: Whatâ€s up with Gallup tracking poll giving Romney a big lead?

The serious problems with Gallup’s application of its likely voter model seemed pretty obvious to me in real time — so they must have seemed obvious to many others. The problems should have seemed obvious to USA Today too, but the paper reported on individual Gallup polls as if they were gospel — and in the process ignored dozens of state-specific polls that gave a much clearer picture of the complexion of the race.

How bad did USA Today end up looking?

Check out the lengthy piece on October 15, 2012 — three weeks before the election: Swing States poll: Women push Romney into lead.

From that breathless article:

Mitt Romney leads President Obama by four percentage points among likely voters in the nation’s top battlegrounds, a USA TODAY/Gallup Poll finds, and he has growing enthusiasm among women to thank.

As the presidential campaign heads into its final weeks, the survey of voters in 12 crucial swing states finds female voters much more engaged in the election and increasingly concerned about the deficit and debt issues that favor Romney. The Republican nominee has pulled within one point of the president among women who are likely voters, 48%-49%, and leads by 8 points among men.

Ha.

Ironically, if Gallup had just headlined its poll of registered voters — a 2 point edge for Obama in both the swing states and nationwide — they would have ended up looking pretty good.

Consider those 12 “swing” states: Colo., Fla., Iowa, Mich., Nev., N.H., N.M., N.C., Ohio, Pa., Va. and Wis.

Three weeks after that poll, Romney won North Carolina by almost exactly 2 points. But he lost all other 11 states. Obama won 8 of those 11 states by 5 points or more, including Michigan with a 9.5 point margin.

In all, Obama ended up winning those 12 states by 4.09 percent, slightly better than his 3.85 percent win nationwide.

By the way, for a great spreadsheet of the final popular vote tallies, check out this spreadsheet by David Wasserman of the Cook Political Report.

There are some obvious lessons here for reporters.

As Nate Silver might say, every poll has the potential to tell you something, but it’s easy to overplay the importance of any single poll.

A poll of 1,023 registered voters across 12 different states — like the one referenced in this piece — is especially problematic because it necessarily means very small samples in some of those states. Indeed, Gallup’s margin of error for the swing state poll reported on so definitively — legible in the fine print — was +/- 4 to 6 points.

It’s simply irresponsible journalism to report so heavily on a single poll when there are dozens of others being done each week.

It’s even more irresponsible when the poll in question so obviously contradicts state level polling with much larger sample sizes in each individual state.

And it’s still more irresponsible to report so heavily on a single poll when numbers crunchers like Nate Silver at FiveThirtyEight and Sam Wang of the Princeton Election Consortium are arriving at such wildly different conclusions based on all the numbers in the public record.

I don’t read USA Today enough to know if they published any sort of mea culpa for this type of coverage, but it’s worth noting that the Washington Post, The New York Times, and other major outlets similarly gave heavy coverage to polls they had commissioned.

Gallup responded to its big miss by missing the point — see Sam Wang here or my response here.

Let’s hope USA Today and other news outlets that commission polls are more measured in their reporting in 2014, 2016, and beyond.

From Talking Points Memo, Ryan: It ‘Looked Like We Stood A Pretty Good Chance Of Winning†On Election Day:

“The polling we had. The numbers we were looking at looked like we stood a pretty good chance of winning,” Ryan said. “So, when the numbers came in, going the other direction. When we saw the turnout that was occurring in urban areas which were really fairly unprecedented, it did come as a bit of a shock. So, those are the toughest losses to have — the ones that catch you by surprise.”

Click here for the entire interview video and read a bit more:

As they saw the states fall one-by-one, reality began to settle in, and by the time Ohio was called for President Barack Obama, they knew the race was over.

“Once we realized that we probably weren’t going to win Ohio, that’s when we realized that it probably wasn’t going to turn out for us,” Ryan said.

Really? Even on election night, with North Carolina too close to call deep into the night, with Obama performing well in Virginia and Florida, with states like Iowa and Colorado showing good returns for Obama — even with all that going on, Ryan thought they were in a position to win until Ohio was called?

Wow, talk about insularity.

Anyway, there’s a bit of a bizarre memo posted on Gallup’s website today. It’s dated November 9th, but no one in the media covering such things seems to have seen it until today. From Polling, Likely Voters, and the Law of the Commons by Gallup Editor-in-Chief Frank Newport:

We will continue to examine our likely voter procedures, a real key to understanding the final popular vote. Our final estimate of registered voters was an unallocated 49% for Obama, 46% for Romney. The transition to likely voters moved that to the unallocated 49% Romney, 48% Obama.

We have modified our likely voter procedures in a number of ways over the years since they were first developed by George Gallup and Paul Perry decades ago. But I think it is clear that voting today is subject to new pushes and pulls, including, in particular, the highly sophisticated ground games employed by the Obama (and, to a lesser degree, the Romney) campaign this year. These methods may in the end affect voters who were not certain about voting at the time of a poll interview, but who were brought into the voting pool at the last minute by aggressive get-out-the-vote and late registration methods. Our traditional “bootstrap” method of identifying likely voters is self-weighting — letting voters†responses to questions determine their probability of voting. This bears investigation. We will use the governmentâ€s post-election data, along with internal evidence, to see if further assumptions, investigations, or changes might be necessary.

We do believe that the presidential campaign underwent significant changes as it progressed this year. Romney clearly gained as a result of the first debate in Denver, and he held onto at least a marginal lead position in our polling until the week before the election, when Superstorm Sandy hit. Obama gained five points on the gap between our last pre-storm polling and the final poll. It may be that he continued to gain on into Election Day.

So Gallup thinks they were right all along? Romney was 6 points up a week before the election, even though the vast weight of the polling gave the edge to Obama? Then Hurricane Sandy swung the election 5 or more points toward Obama, even though its main impact was in states where Obama was already destined to prevail? And that there their miss of a single point (“a statistical tie”) was acceptable given that Obama’s win will be more than 2.5 points once all the votes in California are counted?

Sitting here in Savannah, I was convinced that Gallup had dramatic problems with its likely voter model, which I wrote about in October and then referred to briefly in my electoral prediction on Nov. 5th:

Newport also takes a jab at Silver and other aggregators, while raising a legitimate concern:

Itâ€s not easy nor cheap to conduct traditional random sample polls. Itâ€s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others†polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other peopleâ€s polls in the next election cycle and cut out their own polling.

Silver has already tweeted a couple of reasonable and pithy responses:

Gallup’s demographic weighting algorithms probably contain a few bugs. They need to hire better statisticians.

— Nate Silver (@fivethirtyeight) November 13, 2012

Polls shouldn’t bounce around as much as Gallup’s, given their large sample sizes. Something is designed badly and introducing noise.

— Nate Silver (@fivethirtyeight) November 13, 2012

Sort of a disappointing response from Gallup to all of this.

But that’s far out of step with the general weight of the polling. Real Clear Politics’ average of the polls, which includes Romney’s big Gallup edge, shows Obama with a .1 percent edge. Nate Silver’s much more complex model at FiveThirtyEight shows Obama with a 1.1 percent edge right now.

As it turns out, there are plenty of reasons not to trust Gallup’s numbers in its likely voter model.

From Ezra Klein’s About that Gallup poll: Is Romney really up by 7? And will Obama win the election anyway?:

Dig into the poll, and youâ€ll find that in the most recent internals theyâ€ve put on their Web site — which track from 10/9-10/15 — Obama is winning the West (+6), the East (+4), and the Midwest (+4). The only region heâ€s losing is the South. But heâ€s losing the South, among likely voters, by 22 points. Thatâ€s enough, in Gallupâ€s poll, for him to be behind in the national vote. But itâ€s hard to see how that puts him behind in the electoral college.

So if you accept the Gallup tracking poll as definitive, as Jennifer Rubin at The Washington Post rather lamely did today, then you’re embracing a nightmarish scenario of a popular vote/electoral vote divergence in which Romney gets the most votes but gets beaten handily in the Electoral College. Obama can lose the entire South (including Virginia and Florida), and still win the election, as the map at left shows.

Of course, Gallup’s own numbers are nonsensical. Obama is entirely likely to get swept in the South, but he’s unlikely to lose by more than 10 points in Georgia and is likely to be within a point or two (one way or the other) in Florida. It’s just not plausible that the other states could drag that margin so much higher for Romney.

Also, the divergence between Gallup’s likely voter result (+6 for Romney) and registered voter result (+1 for Romney) reflects an extreme edge — one that seems very unlikely — for Obama among registered voters who won’t vote.

And Klein points to another clear problem with the likely voter model:

“The likely voters model takes into account changes in the response to questions about how closely theyâ€re following and how enthusiastic they are,” [Frank Newport, editor-in-chief of Gallup] said. “Itâ€s not just capturing underlying movement — itâ€s representing changes in enthusiasm.”

There was certainly a surge in enthusiasm among Romney supporters after the first debate and many dispirited Obama supporters too. But everybody still just gets one vote, and polls have showed movement within a very narrow range for months now.

Also, FiveThirtyEight has pointed out some absurd swings in Gallup’s election polling going back many years. See Gallup vs. the World:

In 2010, Gallup put Republicans ahead by 15 points on the national Congressional ballot, higher than other polling firms, which put Republicans an average of eight or nine points ahead instead.

In fact, Republicans won the popular vote for the United States House by about seven percentage points — fairly close to the average of polls, but representing another big miss for Gallup.

Apart from Gallupâ€s final poll not having been especially accurate in recent years, it has often been a wild ride to get there. Their polls, for whatever reason, have often found implausibly large swings in the race.

In 2000, for example, Gallup had George W. Bush 16 points ahead among likely voters in polling it conducted in early August. By Sept. 20, about six weeks later, they had Al Gore up by 10 points instead: a 26-point swing toward Mr. Gore over the course of a month and a half. No other polling firm showed a swing remotely that large.

Then in October 2000, Gallup showed a 14-point swing toward Mr. Bush over the course of a few days, and had him ahead by 13 points on Oct. 27 — just 10 days before an election that ended in a virtual tie.

As Silver notes in that post, these wild swings aren’t reason to give up on Gallup. It’s likely tracking enthusiasm in ways worth noting, even if not a good predictor of actual outcomes. (Other Gallup products seem to fare much better, however.)

It’s going to be a really close election — likely no more than 2 points separating Romney and Obama. Obama seems to have the edge in both the popular vote and the electoral college, but a lot can happen in the next three weeks.