Running the Numbers: On the Difference Between Film Study, Analytics
At DallasCowboys.com, I took a look at Kyle Wilber’s future in Dallas. In that post, I went off on a tangent about what I believe are the primary problems with “blind” film study:
“Turn on the tape.”
We hear that a lot, right? Why has George Selvie gone sack-less the past couple of games? Just turn on the tape. Why isn’t Tony Romo throwing downfield? Just turn on the tape. What’s Kyle Wilber’s long-term outlook in the NFL? Well, all you gotta do is turn on the tape.
And there’s no doubt that film study is an important part of football, whether it’s game-planning for opponents or scouting NFL rookies.
But as it stands right now, there are a few problems with blind film study.
First, we really have no consensus on “how” to watch film. Two very well-trained scouts can watch the same tape and come away with two very different opinions. We see that all the time in the draft, and it can be problematic.
Second, there’s not always much relevant film to study. Is it really all that helpful to watch a dozen games of a small-school college prospect competing against players half his size? How about a player who was known to be competing through an injury for an entire year? What about a guy like Wilber who is now a couple of years removed from his college days without much NFL experience? Or players making position switches?
When it comes down to it, film isn’t standardized. We can’t just “turn on the tape” and trust what we see because there are just too many variables for the results to be extremely meaningful. And even when everything is as systemized as it can get, different eyes see different things.
That brings us to the third problem with the current state of film study, and the most damning: It’s not falsifiable. That’s a major, major issue because a lack of falsifiability is a telltale sign of something being unscientific, and thus incapable of improvement.
How do you falsify a scout’s claim that a prospect “plays with heart,” “displays savvy,” or “has great hips”? You can’t. Not without analytics.
Look, scouts are really good at what they do. To watch film and accurately grade individual prospects in an environment as chaotic as the football field is awesome. But we need to accept their opinions in spite of the fact that they can’t be falsified. They can’t be improved upon.
Analytics, on the other hand, is built upon a scientific foundation. Through an evolutionary process, bad stats can become good stats. Statisticians are refining their formulas and models all the time, using data to create more accurate forecasts.
But how can bad film study become useful? How can one scout piggyback on the work of another? As it stands right now, that can’t be done. Each scout needs to learn the nuances of watching film, which requires countless hours of dedication. Even then, there’s probably a fairly low ceiling on what he can provide since there’s no scientific foundation on which he can build. Worse, his opinions must be accepted on faith.
Meanwhile, analytics continue to evolve. The scientific nature of stat analysis – the way in which it can advance – makes it scalable in a way that traditional analytics-deprived scouting is not. If the Cowboys are truly a team built upon valuing “the process” over the outcome, analytics absolutely must be embraced.