Kelce and two kids stacked on top of each other in a football uniform.

It is not possible for PFF to do a good qualitative analysis the way they are doing it. They are assigning an omnibus number to a multifaceted construct with serious myriad interaction effects. Their rankings are asking a question that’s inherently leading us away from useful information.

My beef isn’t just about the evaluation. The stats are the stats and I trust that you’ve done them appropriately. It’s that the entire approach, even downstream. Even if you added every single variable available on football reference, we still wouldn’t have truly actionable information. Good outcomes come from schemes that fit with good player traits and the stats on PFR do a poor job at measuring both those things.

The comprehensive evaluation already exists. It’s called film study. Luckily we can access it for free on youtube from former NFL quarterbacks such as JT OSullivan and Kurt Warner. There are also people like Greg Cosell and Brian Baldinger at NFL Network.

Ultimately any omnibus number that tries to rank a QB’s overall play against the other, even if it has a number of variables feeding into an equation with an effect size as an output, will strip valuable context from the discussion. A QB on one team may look terrible in one place but amazing in another or vice versa (see Tua, Geno Smith, and Watson.) Some QBs are exceptional in some areas but severely lack efficacy in others (See Cam Newton). To put it in statistical terms, football is a gnarly mess of difficult to measure interaction effects.

I would advocate for qualitatively assessing traits and only using quantitative assessments as supplements. With any statistical analysis we need to understand the context in which the numbers exist or we’re prone to misinterpretation.

Chasing down a rabbit hole for a single way to rank quarterbacks in 2023, with our currently measured variables, is unlikely to yield a practically valuable outcome.

The removal of YAC as a way to determine QB skill is pretty dicey. Short passes that go for a lot of YAC aren’t necessarily easy throws. Oftentimes the windows are small both in terms of surface area and temporal length. Being able to create space on short passes with a QB’s eyes is also a skill.

Additionally, while saying YAC is largely a function of receiver play and good blocking may have some truth to it, so does CAY. A receiver like Davante Adams or Megatron who can reliably gain separation deep will inflate a CAY just like how a receiver like AJ Brown or Deebo Samuel can inflate YAC.

Offensive line pass blocking skill also systematically biases CAY values. Poor pass blocking lines or average lines who have to face formidable defensive lines will necessarily have game plans to neutralize this issue. You will see a lot of slide protections and screens, which is a choice made somewhat independent of QB skill.

These analyses are interesting, however I would caution using these numbers as a way to rank QBs against each other when there is this amount of noise in the data. This is a theme true for many football statistics, not just CAY.

However, I do see practical value on using CAY on a case by case basis. If there are high CAY values despite conditions being hostile towards it, as determined qualitatively on film (poor pass blocking line play, game plans reliant on shorter throws for more YAC, etc…), then I would argue there is a strong indication of a QB’s deep ball skill. Vice versa also applies for poor deep ball skill.

There’s already a stat that does what you’re saying. It’s called RATE+.