ESPN’s QBR – Color Me Unimpressed

(Note: Check out the new updated version of the article at our new location with a brand new look and feel: Blog-a-Bills)

As many of us know (mostly from having had it hammered into our heads thanks to the fine folks over in ESPN’s marketing department and not through any free will of our own), ESPN recently unveiled a new QB rating system to replace the  flawed passer rating that has been used as an evaluating tool for QB’s since 1971

Given that it could be what one would consider an advanced metric, it seems relevant to post a few thoughts on the topic. (For the record, and to keep this Bills-based, Ryan Fitzpatrick scored a 48.7 out of 100 last year in ESPN’s QBR, good for 17th out of the 33 QBs who qualified last year).

In what we all should consider to be shady at best and malicious at worst, ESPN won’t actually release the exact nature of the formula. Why, you might ask, should we have a problem with that?

Chase Stuart over at pro-football-reference.com puts it best (read his whole post on ESPN’s new rating here):

Unfortunately, ESPN is keeping its formula a secret from the public. There are lots of legitimate reasons for doing this, but that decision makes it impossible to fully criticize the ESPN QBR. It’s possible that there are serious bugs in the formula, but we have no way of knowing or discovering them. It’s hard to get excited about a rating that says “Matt Hasselbeck is better than Sam Bradford because Hasselbeck has a 42.4 rating and Bradford a 41.0.” Well, why is Hasselbeck’s rating because than Bradford’s? Because ESPN says so, of course. That’s just not very convincing. For all the flaws in traditional passer rating or any of the formulas I’ve come up with over the years, you at least know what the flaws are. You can recreate the rankings because you have access to the formula. You can catch errors. You understand why rankings appear the way they do. And you can catch a simple programming mistake that throws off the computer ratings because of human error.

I’m not normally one to buy into conspiracy theories, nor do I particularly have a problem with ESPN as a “big bad guy.” (Although reading Those Guys Have all the Fun  makes you wonder about some of their decision-making process, including The Decision itself). However, Chase raises excellent points regarding the importance of transparency in statistics.

There is nothing at all convincing to me about saying someone’s rating is better than someone else’s because a giant conglomerate has a secret formula (no Big Macs for me either, thank you). In the end, the conflict of interest is just too great for me not to wonder about some shady, behind-the-scenes dealing.

ESPN has a vested interest in seeing the NFL, and by extension, its players, do well. (Especially the marketable ones, which QBs tend to be). If, during one of their games this year, ESPN decides the best way to promote the game is to highlight the two starting QBs (not an infrequent occurrence), what is to stop them from artificially jacking up the QBR on those two QBs?

This isn’t to say we, as the football watching population, would only watch a game because the QBs have great QBRs. But, we’ve seen the myriad of inane ways ESPN (and others) have promoted games before. If they believe in the effectiveness of some of those promos, how are we to truly know that they wouldn’t feel the same about a promo based on their secret QBR? And, if that were the case, wouldn’t it follow to reason they might think fudging some of those numbers might make the games more attractive to us, the unwashed masses they sometimes appear to think we are?

There are positive aspects to the QBR, and it certainly (in my eyes) is far superior to the current passer rating cited most regarding QB performance. Keeping the sauce secret, though, is not the way to gain any sort of credibility in the growing advanced football metrics world.

It does, however, appear to in the fast food world.