The Comey II Effect

January 16, 2017


US presidential elections are extremely complex events. There are a lot of a lot of moving parts from candidate selection to the Electoral College. The occur only every four years and there’s been under 60 in total, with maybe a third in a reasonably modern era (e.g., with mass communication). Furthermore, the US, in general and in the political scene, exhibit complex dynamics. It’s changing all the time! This makes them hard to study. This XCKD cartoon is a good reminder to be epistemicly humble.

However, this doesn’t mean we should just shrug our shoulders. Predicting winners is going to be hard we predict because increasing polarization, plus some structural features, means elections will be close. When elections are close, they are hard to call (at least individually). That doesn’t mean we aren’t right to predict that they will generally be close and that gaps between the popular vote and the electoral vote will increase and tend to benefit Republicans. These seem quite true.

However, autopsy is different that predicting.  When we look backwards, we are dealing with one set of events, not many possible ones. Now, of course, we can’t go back in time and rerun stuff. Plus evidence degrades quickly with time. But we are looking at something static. And by careful study we can learn surprising things! For example, the consensus is that Ross Perot did not spoil the 1992 election for George Bush. See this excellent blog post by Samuel T. Coop that has good links to the literature as well as a nice discussion both of the direct and possible indirect effects of Perot on the 1992 and 1996 elections. Note that the kind of study one does (and confidence one has) depends critically on how one frames the question. “Did Perot spoil the election for Bush?” has one kind of answer if you operationalise it as “Were Perot voters such that they would have voted for Bush if Perot had been removed from the ballot the week before the election in numbers and patterns sufficient to change the election outcome?” A much harder question to answer is, “If Perot never entered the race, would Clinton still have one?”

The former question might nevertheless be unanswerable or difficult to answer if we don’t have good data. The latter question might well be simply unanswerable absent singularity level simulations.

In general, effect questions about events close to an election are more tractable than ones about events further away. This shouldn’t be surprising.

Comey had two big interventions in the election: The first was in July 2016 when he cleared Clinton of wrong doing, closed the election, and (wrongly and inappropriately) bad mouthed her. The second was when he released a letter a week before the election saying, “Hey! We might have something to look at!” (then another letter just before the election saying, “Oops, nothing here”). There is no disputing the wrongness and inappropriateness of the second intervention. The “best” interpretation is that he was trying to get ahead of rogue agents in New York. Of course, this is several levels of failure including that rogue agents shouldn’t be rogue and the way he did things was strongly biased against (a totally innocent!) Clinton. In any case, it was against policy and precedent, and it was definitely biased toward Trump. We can see this in his current refusal to discuss the FBI’s investigation of the Trump-Russia issue with congress even in private. This is the Bush v. Gore “We only intend this discussion/principle to hold for this one case where it steals the election” bit all over again.

One thing that is conclusively established is that Comey did some very wrong things and that Comey’s FBI is in the running for one of the worst FBIs ever (which is saying something).

But not all wrong actions have bad consequences. (Luck can intervene.) A critical question, thus, is whether Comey threw the election to Trump. Scott Lemieux thinks that the answer is yes, partly based on a recent Vox article by McElwee,  McDermott, and  Jordan which looks at 4 pieces of evidence that “the Comey effect was real, it was big, and it probably cost Clinton the election”.

Some Qualifications

Both Scott and the Vox folks try to disarm one of the standard counter arguments, to wit, that even if Comey II had an effect, it was dwarfed/only made possible by the badness of Clinton and/or her campaign. Their attempt is roughly, “Yes, the campaign made mistakes, but all campaigns do.” Meh. I want to see more affirmative evidence that the Clinton campaign was bad other than “it shouldn’t have been close enough for Comey to affect” or even “She lost to Trump.” The overall evidence is that Clinton is a pretty good candidate (nearly won against Obama and won the popular vote in a year when the fundamentals, the press, and the FBI and Russia/wikileaks, etc were against her). In 2008, we had very specific evidence that campaign competency was a problem (critically, they didn’t pay enough attention to caucuses and delegate math). There’s little such evidence this year. So, pfft. She ran a pretty good campaign in adverse circumstances. Qua candidate, she has vulnerabilities, but clearly also strengths: It seems that it took fundamentals, plus the press, plus Russia, plus Wikileaks, plus the FBI to beat her narrowly.

So, meh, to that.

We’re looking for a change in voter behavior from what it would have otherwise been without Comey II but with no new events. Thus, if some hackers held back some Comey II equivalent info because they thought Comey II was sufficient, then the counterfactual “But for Comey II, Clinton wins” is false. But this isn’t the right standard since we are trying to determine the effect of Comey II on the actual election. This is an autopsy. If someone gets hit in the brain by three bullets spaced three seconds apart, each of which were sufficient to kill them, we don’t say that the first bullet wasn’t the cause of death just because if it hadn’t been fired the person would still be dead.

Finally, even if Comey II had no effect, it would still be unjustified and a serious failing on Comey’s part. If it had some effect that doesn’t seem quite enough to throw the election, it’s still very bad. However, obviously, throwing the election aligns intent and horrific effect.

The Vox Case

The conclusion is:

the Comey effect was real, it was big, and it probably cost Clinton the election

The evidence is in four “exhibits” (and I grabbed their headings to make this list):

  1. “Exhibit 1: the state polls.”
    This is a weird one because it should say something like, “Looking at the state polls, we see a decisive, unusually large shift toward Trump in key states. Indeed, the average is from Clinton +3 on the 28th (Comey II day) to Trump +1.2…4 points! in a week!” However, they are focused on the “surprise” aspect of the election and how some polls just before the election undershots Trump’s actual win. I think this is important (see below), but it’s not part of the first order case for the Comey Victory. They are convincing that some states where the polling didn’t show the trend were underpolled. But this is sorta beside the point. The idea is that there was a swing in votes that was captured by a swing in the polls (and between the polls and the actual outcome).
  2. “Exhibit 2: the national polls”
    Basically, every account of the national polls showed a big hit (2-3 points) against Clinton in reaction to Comey II. Comey I also produced a direct swing in the polls against her. So Comey announcement affecting the polls seems reasonable. Note that this does not yet generate a Trump win…national polls still had Clinton up. But, she did win the popular vote!
  3. “Exhibit 3: The early voting numbers compared with the late deciders”
    Clinton led in a lot of the early voting. That could be a biased sample, but  Clinton had a huge drop between early and election day voting was in blue states like RI. Obama saw gains in such circumstances.
  4. “Exhibit 4: media coverage of email, email, and more email”
    EMAILZ!!! dominated the news coverage and, correlatively, voter perceptions of Clinton, “While 79 percent of registered voters had heard “a lot” about Clinton’s emails, only 23 percent heard “a lot” about Trump’s housing discrimination, 27 percent heard “a lot” about the Donald J. Trump Foundation’s illegal political contribution to the Florida attorney general, and, surprisingly, only 59 percent had heard a “a lot” about the Hollywood Access tape.”

Only 59% of voters heard “a lot” about the Trump tape?! Whoa.

We see some undeniable Comey II effects. 4 (media coverage) is just plainly evident. (Note: Obviously the media were complicit. They could have treated Comey II correctly and didn’t. How this absolves Comey is a mystery to me. He knew or should have known what would happen.) 1 & 2 seem probable both based on the timing and on past effects. 3 needs a bit of work to directly establish the relationship, but as a supporting consideration is quite alright. That all these things march together strengthens the story. As the Vox piece puts it:

Instead, the evidence is clear, and consistent, regarding the Comey effect. The timing of the shift both at the state and national levels lines up very neatly with the publication of the letter, as does the predominance of the story in the media coverage from the final week of the campaign. With an unusually large number of undecided voters late in the campaign, the letter hugely increased the salience of what was the defining critique of Clinton during the campaign at its most critical moment.


Let’s recall what has to have happened for a Comey Victory:

  1. Before Oct 28th, Clinton had to have been really ahead. That is, enough people in the right places would have voted for her or, at least, not voted for Trump (e.g., by splitting their ticket or staying home).
  2. On election day, we have the result we have.
  3. People changed their voting behavior (e.g., stayed home, changed their vote, or came out for Trump).

And, of course, this change had to be caused by Comey II. But I think if we can establish 1-3 then we’ve done rather well. Direct polling on this (i.e., “Did Comey II change your voting behavior”) would be welcome but will get less reliable the further out we go (due to recall bias).

So, we have a shift in the polls with the right timing plus a mechanism (coverage). Isn’t this enough?

There are some possible alternatives.

  1. The shift was due to some other factor like awesome Trump ground game. (Unlikely.)
  2. Before or around Oct 28th, Trump was ahead or close to being so and the Clinton lead was a polling illusion.
  3. Trump’s mostly was ahead but there was a strong, consistent polling illusion.

1 suggests a different dynamic. 2 and 3 suggest that the race was more static than a Comey Victory requires.

Could the race have been more favourable to Trump than the polls suggested? Event driven swings in polling, esp. large ones in early polls, have been viewed with skepticism for quite some time. Convention bounces, for example, tend to be bounces: They boost the candidates polls for a short while then fade. The idea that voting intention would be so fickle seems improbable, esp. in this day and age. Recall the myth of the independent voter, that is, in spite of increasing numbers of Americans identifying themselves as independent, only a small fraction are “true” independents, that is, exhibit voting behavior markedly different than some partisan. (And of course, “independent” doesn’t mean “indecisive”.) One rising explanation of such poll volitility without real change is “differential response rates“.

Roughly speaking, given the low response rates typical of modern polling, shifts in polling results can come from a systematic change in who is likely to respond. Note that this is different than standard sampling issues and isn’t addressed by, for example, larger sample sized. It’s also different from problems in your likely voter models. (Assuming your likely voter model is stable, it could give a consistent error but is unlikely to yield big swings.)

Exhibit 3 suggests that the change was real: We have an unusual difference in actual voting behavior from early voting to election day. We have a large number of “undecided” who broke strongly for Trump. So we have a clear causal story.

One challenge with that story is whether the shift from undecided to Trump would have happened anyway. This is similar to the “shy Trump” effect sometimes posited. But basically, people tend to vote identity. Identity doesn’t tend to shift over a campaign, so who people will end up voting for is pretty predictable. But who people think or say they are going to vote for is more flexible. And this isn’t because they are shy (shy people lie) but because they are genuinely conflicted.

Imagine a Bernie supporting, Green curious Democrat. They might well flirt with the idea of voting Green the whole election season but when it comes down to the crunch they vote Clinton because, well, Trump! (I was that sort of Nader supporter in 2000. Even though I though Gore would win and had bought some of the “not a dimes worth of difference” line, I couldn’t risk being part of a Bush win.)

So it is possible that Republicans who were “concerned” by Trump always were going to vote for him. We saw this in a lot of “Never Trumper” elites who declined to support him until they did.

Worse, experientially, many of these folks will take Comey II as the causal reason for their vote. But, the argument goes, that’s a cognitive illusion. In reality, they would have always found a way to vote for Trump. This distinction will be hard to sort out, if even possible.

What Should We Believe About Comey II?

Scott advances a strong (but slippery) version of the Vox conclusion:

As I’ve said before, at this point to deny the effects of Comey’s interventions is essentially trooferism. There is no serious alternative explanation that can account for the data. The “durrrr, correlation is not causation, durrr” argument loses any plausibility when you consider that every Comey intervention caused a wave of negative media coverage about Clinton and was followed by a significant decline in national polls numbers. The “polls can’t account for Trump being a celebrity” response fails to explain why Election Day voters were more affected by Trump’s celebrity status than early voters although he didn’t become more famous in the interim (but people were treated to an obsessive wave of negative coverage about Clinton.) Even if Comey had not sent the letter on October 28, we can be as confident that Clinton would have won as we could ever be confident in such a counterfactual.

The strongest line against it is that voting intentions are fairly stable and we have some mechanisms to explain that polling produced shifts are illusory.  But, let’s not, that there is no direct evidence for this in the current situation. Yes, differential response, shy Trump, and dithery Trump could explain everything, but even for early event responses evidence is thin on the ground. It’s mostly the general thesis that voting intention is stable. And clearly, there’s some truth to that! I was never going to vote for the Republican candidate, ever, in 2016. Ever. There are lots of similar sorts. The general closeness of elections is suggestive as is the research on nominally independent voters.

One interpretation of the election is that campaign quality doesn’t matter at all, given Trumps weak traditional campaign. Specifically, campaign quality doesn’t have a causal effect (as opposed to a balance of causes). However, given Russian hacking, Wikileaks dribbling, Comey, and a compliant press, a rival interpretation is that narrow Republican campaign quality matters less because the Republican campaign includes parts of the US government as well as state and non-state actors as well as the press.

The very closeness of the results in key states is also suggestive. Comey doesn’t have to swing or consolidate a lot of votes in the Clinton firewall to break it. Unlike conventions which are 1) standard, expected events and 2) way early in the campaign before voter intentions have solidified, Comey II was imminent to the election. So it’s more reasonable to suspect real changes rather than differential response.

So, I lean toward the Comey II effect was real and likely made the difference.

Now, there are some definite bias risks here. Confirmation bias (partly due to anchoring) is strong. I’ve been a proponent of a version of the stability hypothesis (though, usually of the “polls far out aren’t super reliable; polls in Nov tend to be” sort). I think differential response rates are fascinating and provide an elegant explanation of convention bounces. (I think all polls should publish their response rates!) I feel Trump is illegitimate on many fronts. Comey’s actions (and the press reaction) are clearly indefensible. So there’s a lot of room for motivated reasoning here.

That being said, it’s clearly possible to swing too far the other way. The evidence for a Comey Steal are more direct and multifaceted. (Early to late voting behavior isn’t subject to polling illusion!) The idea that that coverage had no effect seems pretty bonkers. You’d want some very strong evidence for that.

So, trooferism? Maybe? The Vox picture (however awkwardly put) is pretty compelling. We’ll see how the evidence evolves. It’s still the case that saying, “Pretty compelling, but we still need some details to know how big the Comey effect was” is reasonable, but perhaps on the edge of reason. The “you can’t know!!!!!” folks are clearly way out of line.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: