Now there’s an excellent piece in the Huffington Post about that issue and several others that could have contributed to Gallup’s failings: Gallup Presidential Poll: How Did Brand-Name Firm Blow Election?.

From that piece:

Since the election, the Gallup Poll’s editor-in-chief, Frank Newport, has at times downplayed the significance of his firm’s shortcomings. At a panel in November, he characterized Gallup’s final pre-election poll as “in the range of where it ended up” and “within a point or two” of the final forecasts of other polls. But in late January, he announced that the company was conducting a “comprehensive review” of its polling methods.

There is a lot at stake in this review, which is being assisted by University of Michigan political scientist and highly respected survey methodologist Michael Traugott. Polling is a competitive business, and Gallup’s value as a brand is tied directly to the accuracy of its results.

The firm’s reputation had already taken a hit last summer when an investigation by The Huffington Post revealed that the way Gallup accounted for race led to an under-representation of non-whites in its samples and a consistent underestimation of Obama’s job approval rating, prompting the firm to make changes in its methodology. (Since Gallup implemented those changes in October, the “house effect” in its measurement of Obama’s job rating has significantly decreased.)

It’s a really interesting piece that discusses myriad issues that polling companies are facing in the 21st century.

Highly recommended for those who followed the numbers closely last year. Those numbers were pretty clear if one trusted the data. Ironically, Gallup didn’t trust its own data enough. If they had reported their registered voter polling versus their likely voter polling, they would have been considered one of the best performers of the election cycle.

Gallup will no longer be conducting polls for USA Today, the two organizations announced Friday.

Both said the split, after 20 years of collaboration, was a mutual decision based on the changing media and polling landscape.

“Given these shifts, Gallup and USA Today have made a mutual decision to move in independent directions beginning in 2013, and Gallup will evolve the polling it conducted in partnership with USA Today in some different and new strategic directions,” Gallup said. “As it has been, Gallup.com will remain the primary source for Gallup polls conducted in the U.S. and around the world.”

USA Today said in its own statement that it is already in negotiations with another pollster.

Well let’s hope that whatever pollster USA Today ends up partnering with in upcoming elections, the newspaper has at least learned some key lessons from 2012. It doesn’t look like Gallup has learned any.

I wrote a number of posts critical of Gallup both before and after the general election in November. In mid-October, reading the work of Nate Silver and others, I even asked: Whatâ€s up with Gallup tracking poll giving Romney a big lead?

The serious problems with Gallup’s application of its likely voter model seemed pretty obvious to me in real time — so they must have seemed obvious to many others. The problems should have seemed obvious to USA Today too, but the paper reported on individual Gallup polls as if they were gospel — and in the process ignored dozens of state-specific polls that gave a much clearer picture of the complexion of the race.

How bad did USA Today end up looking?

Check out the lengthy piece on October 15, 2012 — three weeks before the election: Swing States poll: Women push Romney into lead.

From that breathless article:

Mitt Romney leads President Obama by four percentage points among likely voters in the nation’s top battlegrounds, a USA TODAY/Gallup Poll finds, and he has growing enthusiasm among women to thank.

As the presidential campaign heads into its final weeks, the survey of voters in 12 crucial swing states finds female voters much more engaged in the election and increasingly concerned about the deficit and debt issues that favor Romney. The Republican nominee has pulled within one point of the president among women who are likely voters, 48%-49%, and leads by 8 points among men.

Ha.

Ironically, if Gallup had just headlined its poll of registered voters — a 2 point edge for Obama in both the swing states and nationwide — they would have ended up looking pretty good.

Consider those 12 “swing” states: Colo., Fla., Iowa, Mich., Nev., N.H., N.M., N.C., Ohio, Pa., Va. and Wis.

Three weeks after that poll, Romney won North Carolina by almost exactly 2 points. But he lost all other 11 states. Obama won 8 of those 11 states by 5 points or more, including Michigan with a 9.5 point margin.

In all, Obama ended up winning those 12 states by 4.09 percent, slightly better than his 3.85 percent win nationwide.

By the way, for a great spreadsheet of the final popular vote tallies, check out this spreadsheet by David Wasserman of the Cook Political Report.

There are some obvious lessons here for reporters.

As Nate Silver might say, every poll has the potential to tell you something, but it’s easy to overplay the importance of any single poll.

A poll of 1,023 registered voters across 12 different states — like the one referenced in this piece — is especially problematic because it necessarily means very small samples in some of those states. Indeed, Gallup’s margin of error for the swing state poll reported on so definitively — legible in the fine print — was +/- 4 to 6 points.

It’s simply irresponsible journalism to report so heavily on a single poll when there are dozens of others being done each week.

It’s even more irresponsible when the poll in question so obviously contradicts state level polling with much larger sample sizes in each individual state.

And it’s still more irresponsible to report so heavily on a single poll when numbers crunchers like Nate Silver at FiveThirtyEight and Sam Wang of the Princeton Election Consortium are arriving at such wildly different conclusions based on all the numbers in the public record.

I don’t read USA Today enough to know if they published any sort of mea culpa for this type of coverage, but it’s worth noting that the Washington Post, The New York Times, and other major outlets similarly gave heavy coverage to polls they had commissioned.

Gallup responded to its big miss by missing the point — see Sam Wang here or my response here.

Let’s hope USA Today and other news outlets that commission polls are more measured in their reporting in 2014, 2016, and beyond.

Regular readers know that I was arguing even before the election that the likely voter models were screening out far too many Americans who were likely in fact to vote. That became more and more obvious as we lumbered toward election day.

At a forum last week at Harvard, Daniel Axelrod and others spoke about campaign strategy. From The Washington Post piece, regarding the Obama campaign’s amazingly high level of nightly polling:

The reason [they were so confident]: Massive amounts of their own polling — not just nationally and in individual states, but in nightly surveys of 9,000 likely voters across 10 battleground states.

“All three of them were saying the same thing . . . in a way that gave us real confidence,” campaign manager Jim Messina said in remarks released Monday. “We thought we knew exactly where the electorate was.”

Another thing they learned, said Obama strategist David Axelrod, was that as many as 20 percent of the people who were actually going to vote were not being picked up as “likely voters” in publicly available surveys.

“It just becomes a big horse-race story, and you guys donâ€t even know where the horses are,” Axelrod said.

Time for Gallup and other organizations to revisit their likely voter models, as I’ve said repeatedly.

ORIGINAL POST:

Another dry, wordy, must-read from Nate Sliver at FiveThirtyEight about the most puzzling question of campaign 2012: Why were Romney and his team so surprised by the election night results, when statistical models showed them trailing significantly in the states that mattered most?

Check out When Internal Polls Mislead, a Whole Campaign May Be To Blame.

The main takeaway:

Nonetheless, the seeming inaccuracy of Mr. Romneyâ€s internal polls ought to present a warning to future campaigns. The problems with internal polls may run deeper than the tendency for campaigns to report them to the public in a selective or manipulative way. The campaigns may also be fooling themselves.

Silver goes through the numbers, some based on Noam Scheiber’s reporting at The New Republic: Exclusive: The Internal Polls That Made Mitt Romney Think He’d Win.

Assuming Scheiber really did see some of the Romney campaign’s internal polls, the differences from reality are striking. The internal poll had Romney up 2.5 in Colorado and 3.5 in New Hampshire, for example. He lost those states by 5.4 and 5.6, respectively. Those differences are hard to account for by any logical measure.

I wonder if the Romney campaign systematically “unskewed” the polls to match a predetermined — and erroneous — notion that whites would surge to the polls and massive numbers of minorities would stay home?

The most interesting parts of Silver’s latest post are the discussions of the politics of internal polling — the decisions to selectively release or to preview on background certain numbers for gullible reporters who assume that the campaigns have the real numbers. Here:

But when campaigns release internal polls to the public, their goal is usually not to provide the most accurate information. Instead, they are most likely trying to create a favorable news narrative — and they may fiddle with these assumptions until they get the desired result.

The Democratic pollster Harrison Hickman, who testified under oath in the federal case against John Edwards, put this bluntly, describing the release of internal polls to the news media as a form of “propaganda”:

Hickman testified that when circulating the polls, he didnâ€t much care if they were accurate. “I didnâ€t necessarily take any of these as for — as you would say, for the truth of the matter. I took them more as something that could be used as propaganda for the campaign,” the veteran pollster said.

In keeping with history, the Democrats did exceedingly well with black voters — better than 90 percent. But that’s only slightly better than Democratic candidates in recent history.

Obama also got over 70 percent of Hispanics, a small but growing portion of the electorate. He did even better — about 3 to 1 — among the small but rapidly growing Asian American population (David Brooks has an interesting column about that).

Obama also dramatically outperformed Romney among the 5 percent of exit poll respondents who identified themselves as gay. From the NYT’s Gay Vote Seen as Crucial in Obamaâ€s Victory (written by Micah Cohen, btw, a regular contributor to FiveThirtyEight):

But the backing Mr. Obama received from gay voters also has a claim on having been decisive. Mitt Romney and Mr. Obama won roughly an equal number of votes among straight voters nationwide, exit polls showed. And, a new study argues, Mr. Romney appears to have won a narrow victory among straight voters in the swing states of Ohio and Florida.

Mr. Obamaâ€s more than three-to-one edge in exit polls among the 5 percent of voters who identified as gay, lesbian or bisexual was more than enough to give him the ultimate advantage, according to the study, by Gary J. Gates of the Williams Institute at the U.C.L.A. School of Law, in conjunction with Gallup. The results are consistent with earlier research on the size and political beliefs of gay voters.

Gay voters are another demographic group — along with African-Americans, Latinos, Asian-Americans and Jews — in which Democrats have been winning big victories over Republicans. Some of the groups are small, but together they make up about one-third of the electorate, forcing Republicans to win large victories among the remaining two-thirds of voters to win elections.

There’s also a growing number of men and women telling exit pollsters that they are gay — 6.4 percent of those from 18 to 29 compared to just 1.9 percent of those over 65.

Obama’s statements in support of same sex marriage have certainly helped his standing with gay voters, who also tend to take other socially liberal positions. But there are plenty of gay voters who hold libertarian or fiscally conservative views who could conceivably be convinced to vote Republican, no matter a politician’s stand on a single issue like gay marriage. Also, it’s worth noting that gays are also members of other groups — white, black, Asian, etc.

Almost certainly, some of the Republican establishment will try to tailor future policy positions to appeal to specific blocs of voters. It will be interesting to see if that works, or if Republican social policies will need to be overhauled dramatically to appeal to these various minority groups.

And if the party’s positions on social issues become more libertarian, what will happen to the base of social conservatives?

Lots of interesting trends to watch.

From Talking Points Memo, Ryan: It ‘Looked Like We Stood A Pretty Good Chance Of Winning†On Election Day:

“The polling we had. The numbers we were looking at looked like we stood a pretty good chance of winning,” Ryan said. “So, when the numbers came in, going the other direction. When we saw the turnout that was occurring in urban areas which were really fairly unprecedented, it did come as a bit of a shock. So, those are the toughest losses to have — the ones that catch you by surprise.”

Click here for the entire interview video and read a bit more:

As they saw the states fall one-by-one, reality began to settle in, and by the time Ohio was called for President Barack Obama, they knew the race was over.

“Once we realized that we probably weren’t going to win Ohio, that’s when we realized that it probably wasn’t going to turn out for us,” Ryan said.

Really? Even on election night, with North Carolina too close to call deep into the night, with Obama performing well in Virginia and Florida, with states like Iowa and Colorado showing good returns for Obama — even with all that going on, Ryan thought they were in a position to win until Ohio was called?

Wow, talk about insularity.

Anyway, there’s a bit of a bizarre memo posted on Gallup’s website today. It’s dated November 9th, but no one in the media covering such things seems to have seen it until today. From Polling, Likely Voters, and the Law of the Commons by Gallup Editor-in-Chief Frank Newport:

We will continue to examine our likely voter procedures, a real key to understanding the final popular vote. Our final estimate of registered voters was an unallocated 49% for Obama, 46% for Romney. The transition to likely voters moved that to the unallocated 49% Romney, 48% Obama.

We have modified our likely voter procedures in a number of ways over the years since they were first developed by George Gallup and Paul Perry decades ago. But I think it is clear that voting today is subject to new pushes and pulls, including, in particular, the highly sophisticated ground games employed by the Obama (and, to a lesser degree, the Romney) campaign this year. These methods may in the end affect voters who were not certain about voting at the time of a poll interview, but who were brought into the voting pool at the last minute by aggressive get-out-the-vote and late registration methods. Our traditional “bootstrap” method of identifying likely voters is self-weighting — letting voters†responses to questions determine their probability of voting. This bears investigation. We will use the governmentâ€s post-election data, along with internal evidence, to see if further assumptions, investigations, or changes might be necessary.

We do believe that the presidential campaign underwent significant changes as it progressed this year. Romney clearly gained as a result of the first debate in Denver, and he held onto at least a marginal lead position in our polling until the week before the election, when Superstorm Sandy hit. Obama gained five points on the gap between our last pre-storm polling and the final poll. It may be that he continued to gain on into Election Day.

So Gallup thinks they were right all along? Romney was 6 points up a week before the election, even though the vast weight of the polling gave the edge to Obama? Then Hurricane Sandy swung the election 5 or more points toward Obama, even though its main impact was in states where Obama was already destined to prevail? And that there their miss of a single point (“a statistical tie”) was acceptable given that Obama’s win will be more than 2.5 points once all the votes in California are counted?

Sitting here in Savannah, I was convinced that Gallup had dramatic problems with its likely voter model, which I wrote about in October and then referred to briefly in my electoral prediction on Nov. 5th:

Newport also takes a jab at Silver and other aggregators, while raising a legitimate concern:

Itâ€s not easy nor cheap to conduct traditional random sample polls. Itâ€s much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others†polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other peopleâ€s polls in the next election cycle and cut out their own polling.

Silver has already tweeted a couple of reasonable and pithy responses:

Gallup’s demographic weighting algorithms probably contain a few bugs. They need to hire better statisticians.

— Nate Silver (@fivethirtyeight) November 13, 2012

Polls shouldn’t bounce around as much as Gallup’s, given their large sample sizes. Something is designed badly and introducing noise.

— Nate Silver (@fivethirtyeight) November 13, 2012

Sort of a disappointing response from Gallup to all of this.

As most of you probably know, Silver’s method of aggregating polling data was vindicated on Tuesday night, when his model had Obama the favorite in every state that he in fact won. In FiveThirtyEight’s projection of the popular vote, Obama was a 2.5 point favorite. He won the popular vote by 2.6 points.

In a post last night, Silver deals with the basic issue: Which Polls Fared Best (and Worst) in the 2012 Presidential Race.

A snippet:

As Americans†modes of communication change, the techniques that produce the most accurate polls seems to be changing as well. In last Tuesdayâ€s presidential election, a number of polling firms that conduct their surveys online had strong results. Some telephone polls also performed well. But others, especially those that called only landlines or took other methodological shortcuts, performed poorly and showed a more Republican-leaning electorate than the one that actually turned out.

Twenty three polling firms conducted 5 or more polls in the final three weeks of the election. Of those, TIPP had an average error of .9 percent and the tiniest of partisan leans with a .1 percent bias toward Romney. In other words, simply averaging the TIPP polls of the presidential race would have led someone to a prediction as good as Silver’s. But in terms of average error for individual polls, the second best was Google Consumer Surveys at 1.6, with RAND Corporation at 1.8.

But it’s even more interesting to look at the terrible misses of the 60 final Rasmussen polls, which included the tracking polls, and Gallups abysmal performance in its tracking poll. The Washington Post/ABC News poll had a bad year too:

Back on October 19th, I wrote a lengthy post about why Gallup’s numbers didn’t look trustworthy this year.

Time for some soul-searching at Gallup and at Rasmussen too — 60 polls with an average miss of 4.2 percent! With a Republican bias of 3.7 percent! One could argue in Rasmussen’s defense that the race moved toward Obama at the end. But if it did so, it couldn’t have been more than a point in the final week or so.

Also an interesting note: of the 23 polling firms that were most active in gauging the presidential race, 19 of them had a statistical bias toward Romney — the polls that were supposedly biased against him actually turned out to be biased against Obama.

Of 13 national polls released on Sunday, Obama had the lead in 9 of them while 4 were tied. A tie, in this case, is probably good news for Obama, because he does seem to have an edge in the Electoral College math. As Silver has noted, the polls would have to be systematically biased for some reason(s) for so many of them to be wrong.

In fact, I suspect that the likely voter models employed by certain polling firms — including major ones like Rasmussen, Pew, and Gallup — have been too rigorous in applying their likely voter models and have thus understated Obama’s support on election day. Lots of people on the left and right assumed that the Democratic vote would be dampened significantly, especially among minorities, and that the Republican vote would be elevated by anti-Obama fervor. But it looks to me like both of those possibilities have been exaggerated. Of those three polling firms, the slightly rightward leaning Rasmussen has the race as too close to call, Gallup suspended its tracking poll because of Hurricane Sandy (but had shown implausibly large leads for Romney after applying its likely voter model), and Pew has Obama with a 3 point lead among likely voters and a 7 point lead among all registered voters.

So there’s obviously room for Romney supporters to hope, especially if we cherrypick a couple of major national polls and ignore trends in specific states.

Here’s my guess: I think the turnout will be slightly better than the consensus predictions for Democrats generally and African Americans specifically. That will push Obama easily past 270, even though he’s likely to get only about 51 percent of the vote and to lose at least a couple of states that he took in 2008 — Indiana and North Carolina.

Here’s what I’m predicting:

Silver’s a wordy but clear writer; his analyses of the current state of the presidential race are some of the best — and most understated — that you’ll find anywhere.

Of course, he’s better known for his statistical model than for his analysis.

As I hope most of you know, Silver aggregates ALL available polling data, combined with some other economic and demographic data, to put the likelihood of outcomes in specific percentage terms.

In a post on Tuesday, Silver yet again made a compelling case for looking at state polling for the best guide to the election. It’s been a conundrum: national polls have consistently shown a virtually tied race between Obama and Romney for the last few weeks, but the state level polling indicates that Obama is a pretty heavy favorite to win the Electoral College. It’s possible that the national polls and state polls could both be right on the mark — huge Romney margins in populous states like Texas and Georgia could boost his standing in national polls without helping him in the Electoral College.

But it seems likely that the national polls are having more trouble getting good data than state polls. Gallup relies on a likely voter model that makes it bizarrely erratic. Rasmussen has a slight partisan lean, according to Silver. All the polling firms are dealing with low response rates and an increasingly cellphone-dependent culture. It’s a muddle, that’s for sure.

The state level polling is consistently showing Obama headed for something around 300 electoral votes. Obama is apparently ahead in Ohio, which would seem almost certain to guarantee him the election, but he also has other paths to 270 with states where he apparently has even narrower leads: New Hampshire, Iowa, Colorado, and Virginia. Florida looks very close too, but an average of state polling puts Romney slightly ahead.

Is it possible that a wide variety of polls in 7 or 8 key states are showing systemic bias against Romney and/or toward Obama? That just seems really unlikely, given the accuracy of polling averages tracked by Silver in 2008 and 2010.

At the moment, we’re headed toward a map that will look something like this: