Cowboys Film Study: 2nd Down Play-Calling Revisited
A few weeks ago we examined offensive coordinator Jason Garrett’s play-calling trends on 2nd down. Our results shocked us perhaps more than any we have gathered this offseason.
The graph to the left displays Garrett’s tendencies. Notice the disproportionate amount of times Garrett called a run play on 2nd down after a pass play, and vice verse. On 2nd and 3 to 7, for example, Garrett dialed up a run just 29.5 percent of the time following a run on 1st down. In the same exact situations, though, he called a run 76.5 percent of the time after 1st down passes.
Clearly a coordinator’s play-calling tendencies should not be based solely on the previous play-call (regardless of that play’s result). We concluded Garrett fell victim to the idea that “alternating creates randomization.” In his attempt to “mix it up,” Garrett actually became incredibly predictable with his calls. True randomness has no regard for previous happenings. As we have shown, however, Garrett allowed previous plays (not simply the result, but whether it was a run or a pass) to affect his current call.
After publishing that study, we received some criticism that our stats were meaningless without knowing the tendencies of other play-callers from around the league. These criticisms, though, are unjustified.
We are not simply analyzing the percentage of run or pass plays in certain situations. If that was the case, then yes, we would need to know league-wide tendencies to draw meaningful conclusions about Garrett’s own trends.
Instead, we are analyzing the percentage of runs/passes after a certain type of play. Let’s look again at the above graph. On 2nd and 3 to 7, Garrett was 2.95 times more likely to run after a 1st down pass than after a 1st down run. We are not critiquing how often Garrett called a run in general during those situations–that information is meaningless to us.
Since the down and distance on 2nd down is exactly the same regardless of the 1st down play-call, we would expect a truly random play-caller to dial up a run after a 1st down pass the same percentage of the time as after a 1st down run, regardless of what that specific percentage may be. Thus, it is the overall run/pass percentage that would require the knowledge of others’ play-calling tendencies to be meaningful, but not the percentage of runs/passes in a specific down and distance following a specific type of play.
Nonetheless, we were still curious as to the play-calling trends of other coordinators in similar situations. We had a feeling that, because humans perform so poorly in generating random sequences, we would see that others fall victim to the same fallacy as Garrett, i.e. that “mixing it up” will produce randomness.
Of course, it would be impossible for us to study film on every 2nd down play for every team for the entire 2009 season. Luckily, we came across similar statistics on AdvancedNFLStats.com (a tremendous site that we highly recommend). The numbers are listed just above.
The data consists of 14,384 plays, so the sample size is obviously large enough to draw meaningful conclusions. During those plays, teams ran approximately 50 percent of the time after a 1st down pass, but just 28 percent of the time after a 1st down run.
We contrasted these results with Garrett’s 2009 2nd down play-calls (shown to the left). Notice that Garrett calls a 2nd down run after a 1st down run at basically the exact same rate as other coordinators around the NFL. His 2nd down run ratio after a 1st down pass is also incredibly similar to the league-wide average (54.3 percent to approximately 50.0 percent).
So, is this evidence that Garrett is justified in his play-calling? Not at all. Remember, opposing offensive coordinators are not involved in a zero-sum game (meaning the success of one does not necessarily cause the failure of the other). Offensive coordinators around the league can collectively perform well, or collectively do poorly. In the case of 2nd down play calls, it is the latter.
Further, not all teams suffer from this randomization fallacy at the same rate. The Pittsburgh Steelers, for example, have done a tremendous job or randomizing their plays over the last few years (graph below). Notice that their 2nd down run/pass ratio is nearly the same after a 1st down run as it is after a 1st down pass. The closer these bars come to matching, the closer a team is to reaching the Nash equilibrium, and the more successful they will be on offense.
Thus, the failures of other coordinators around the NFL do not justify the failures of Garrett. His success is independent of that of other offensive coordinators. As we wrote in our initial study of this topic, “If the Cowboys want to maximize the productivity of their potentially explosive offense, Garrett is the first person that needs to change. Unfortunately, if his play-calling does not become less predictable, neither will the team’s fate in the playoffs.”
We aren’t backing off from that statement.