Hello after a really long hiatus!
I frequent Giants message boards and there’s a big debate going on whether the Giants messed up by picking running back Saquon Barkley over a highly touted quarterback prospect like Sam Darnold. The argument goes that a quarterback is a more valuable position and that it’s very difficult to find a quarterback with a non-premium pick. I decided to look into the second part of that claim.
Here’s my methodology:
I took a look at all QBs drafted from 1993-2016 that made the Pro Bowl. I chose 1993 because 1992 and prior there was a significant difference in the number of players drafted. I chose a cutoff for 2016 since it’s still too early to analyze the 2017 class. I chose the Pro Bowl because I figure that’s the most objective, albeit not a perfect way to see if a QB is good. In order to filter out flukes, I further filtered out QBs that have only had 1 Pro Bowl. That created a problem for someone like Carson Wentz who has 1 Pro Bowl in 2 years but will get plenty more Pro Bowls (unfortunately for us Giants fans) but has just begun his career. So I looked at the average amount of Pro Bowls per seasons played for all of the QBs with 2 or more Pro Bowls and saw that these QBs on (weighted) average had Pro Bowls in 37% of their seasons. I originally thought to include anyone that had 1 Pro Bowl in 3 seasons even though 33% is less than 37%, I figured it was close enough. In the end, I felt like the talent of QBs that made 1 Pro Bowl in 3 years didn’t warrant it and I made the cutoff 37%, so 1 Pro Bowl in 1 or 2 seasons. This way QBs like Carson Wentz, Jared Goff, and Dak Prescott make it. It’s worth noting that my filtering process isn’t perfect since guys like Matt Schaub who somehow made 2 Pro Bowls got included. I decided to keep those types of players anyway.
I then broke down the data into years and looked up how many Pro Bowl QBs were drafted per year. I further broke down the data by top 10 and not top 10. Not top 10 included undrafted players like Tony Romo. I counted them as if they were drafted within the non-top 10 picks to keep consistency. I then divided non-top 10 Pro Bowl QB draft picks by top 10 Pro Bowl QB draft picks to get the actual ratio. I adjusted for any year where there were 0 QBs in the Pro Bowl from the top 10 and more than 0 afterward (to avoid dividing by 0). All I did was just add both numbers by 1 and calculate the same ratio (e.g. 94 had 0 Pro Bowlers in the top 10 but 2 after, I changed it to 1 and 3 and the ratio became 3/1=3). Then I calculated an expected value which was basically the total amount of non-top 10 draft picks divided by the top 10 draft picks. This is the ratio of QBs we would expect to see if the rate of Pro Bowl QBs kept constant after the top 10. Finally, I created an index by dividing the actual by the expected ratios calculating the rate as a certain percentage of the expected rate. For example, in 1993 there were 224 picks, 10 in the top 10 and 214 thereafter. We expect a there to be 21.4 Pro Bowl QBs to be drafted after the top 10 for every QB drafted in the top 10 if the ratio were to remain the same. However, there were 2 Pro Bowl QBs drafted after the top 10 and 1 in the top 10. So the ratio was actually 2:1 or 2. I took 2 divided it by 21.4 and the resulting index was about 0.09. In other words, the rate at which Pro Bowl QBs were actually selected outside of the top 10 was 9% of expected. I then averaged this index for all the years in the data set, resulting in a 0.05 index. Not very high.
Please see below for my calculation:
One thing you may notice is that not every year is included, I started by only including the years from 93-16 that had at least one Pro Bowl QB drafted (or undrafted). Later on, I will include the years in that range that had no Pro Bowl QBs drafted.
However, I figured that it’s a bit unfair to do the whole draft since you can have non-premium picks as early as the 2nd round. So I decided to look at the earliest round with all non-premium picks and where most if not all teams will get a chance to pick a QB with a non-premium pick. I followed the same process but instead of including all non-top 10 picks+undrafted players, I included non-top 10 picks in the first two rounds (usually nowadays it’s 54 picks). The index to my surprise less than doubled to 9%. Please see below:
Finally, as I briefly mentioned it may be worth including the draft years from 93-16 that no Pro Bowl QB was drafted. The lack of data is still data. I added these as a ratio of 1:1 much like in the previous situation where we would divide by 0. The analysis that included the whole draft remained relatively stable but to my initial surprise the 2 round analysis actually went up rather than down (to 0.13). However, looking more closely it makes sense, there are is not a huge pool of Pro Bowl QBs drafted in the first two rounds, not in the top 10. The years when this is the case result in a 0 ratio and index. The 1 ratios lifted the average. This does bring up the issue of sample size. Below are screenshots for both analyses.
The conclusion is that it is indeed difficult to find a QB with non-premium picks.
Some concerns: From 93-16 there are 33 Pro Bowl QBs that fit my criteria, 16 from the top 10, 17 not from the top 10, and a mere 6 from the top 2 rounds not in the top 10. It’s entirely possible that the sample sizes are too small to come to any conclusions. In addition, I have been criticized for not using All-Pro selections, because the Pro Bowl isn’t always a great indication of a player’s ability. However, I believe All-Pro selections are too restrictive.
My Sources are:
Number of Picks in Each Draft*
* In case there’s confusion, Mr. Irrelevant is the last pick of the draft. This is the best way to look at how many picks there are in each draft on one page and without having to browse through every draft’s Wikipedia page.
**The link is only to the 2016 NFL draft. In order to see what the draft picks are each year, it’s necessary to go to each year’s NFL draft page. I looked at these to see what pick was the last pick of the second round. In theory, I didn’t even need the previous link since the NFL draft pages have that info, but if someone wants to do research on the entire draft only, the last link has it all in one place.
Igor Zolotusky's Analysis 