Mock Drafts: A Thorough Evaluation

This is the introduction to the NFL Draft model I've built using publicly-available mock drafts. The model has significantly out-performed typical mock drafts, and has the capability to do a bunch of cool functions: mapping the probability distribution for when a specific player is drafted, uncovering the players who may be reaches/drops, mapping the odds a particular player lasts until a particular pick, etc. All those cool functions will be shown in part two, later this week, once the mocks become more robust and the top few draft selection are (more or less) set in stone.

First, however, we must evaluate the data that is going to be used; knowing your input data is an extremely important part of building any model. For those of you unaware, I was that douche guy who made the Padres: The Sad Truth documentary that has spawned the Change the Padres movement. In addition to tabulating what the Padres' ownerships/executives have done/said, I have also kept diligent track of many mock drafts over the years. In the sections to follow, I'll break down the various mock drafters I've kept tabs on, and present some key trends seen in mock draft data. (Note: when the model was built, I did not examine 100% of the data...this is standard practice. Below, however, are some various metrics applied to 100% of the data).

How to evaluate a mock draft

Let's start with how NOT to evaluate a draft. Most places that grade drafts, like http://www.thehuddlereport.com/, use the following system, or one similar to it:

• award 1 point for each first round pick who is drafted in the first round.
• award an additional point for each pick who ends up on the correctly predicted team

This system is flawed for several reasons:

• If a team has multiple draft picks in the first round, you'll get a bonus point if the team uses their second selection on the player you actually projected to be selected first. This ends up with some very poor results: for example, anyone who projected the Browns to select Brady Quinn with their first selection in 2007 is awarded two full points for the selection, since Quinn ended up a Brown at the end of the first. Obviously, projecting Quinn to be the Browns first selection in the draft was a horrible projection, but in this system it is considered the best.
• It doesn't take into consideration the degree to which a projection is wrong. If a player mocked at slot 10 goes 11th, it is graded the same as a player mocked at slot 10 going 32nd.
• It doesn't take into consideration that missing by a certain number of slots early in the draft is less excusable than missing by that same number of slots later in the draft. (The early picks are easier to project.)

So, how have I graded the drafts? In the most shortened way I can describe it: the p between where a player is projected to go and where he ends up going is normalized against the (smoothed) typical error seen in that draft slot. In other words: if you miss by 5 selections at a slot where there is generally less error, you'll be more penalized than missing at a slot where there is generally more error. Projecting someone to go later than he does is less rare than projecting someone to go sooner than he does, so this is also accounted for in a subtle way (which would bore everyone if I went into further detail).

Just know this for when I discuss the best mockers below: the more negative the score, the better the draft; a value of 0 is average; a positive value is poor.

The best mock drafters

In what I'm sure will be very disappointing to everyone, the best mock drafters of the 30+ that have been graded since 2007 are Mike Mayock, Mel Kiper Jr., John Clifford, and Todd McShay. And it isn't really close. Since 2007, Mike Mayock's typical score is -8.5, Kiper's is -6, McShay -5.5, and Clifford -5. Robby Esch is the top of the next tier (-2.5). The most run-of-the-mill drafter -- ie. the one closest to 0 -- is Peter Schrager from MSN/Fox Sports. I would tell you the worst mock drafter, but I feel bad publicly goading people for doing a poor job. Just kidding. The worst mock drafter examined is Jeff Mullen, who was just as bad (+8) as the best in the business was good. (Some of his picks last year -- Jonathan Martin 13th, Courtney Upshaw 16th, Mike Adams 23rd, and his 29th/30th combo of Konz/Zach Brown -- were particularly poor.)

All these negative and positive scores are tough to visualize, so here's some visualizations for everyone. The x-axis is the player's actual draft position, while the y-axis is the player's mocked draft position: the tighter the distribution, the better the mock drafter. (Note: The different marks represent different years, but that's not entirely important. Some notable misses are labeled. A default value of 40 is applied when a mock drafter did not rank a particular player.)

Let's start with the best of the best: Mike Mayock, whom you'll notice has no real horrible misses.

Mel Kiper Jr., who has more reaches than Mayock, but less players that he failed to project highly:

Todd McShay, who has just a few more misses than Kiper:

Jeff Mullen and his Jackson Pollock painting:

The raw numbers

Now lets take a look at some aggregate statistics on how accurate, since 2007, the mock drafters have been with projected top ten picks, split by offense and defense. (Average is the mocked average where an unranked player is given a default of 40; St. dev. is the standard deviation of that player's mock position; average difference is the actual draft position minus the mocked draft position; average error is the absolute value of actual draft position minus mocked draft position.)

Position	Number	Average	St. Dev.	Avg. difference	Avg. error
Offense	28	4.53	1.32	0.94	2.05
Defense	20	5.70	1.60	0.00	1.08
QB	11	3.27	0.88	1.72	2.21
Off. w/o QB	17	5.34	1.60	0.43	1.95

And players projected to go between 10 and 20:

Position	Number	Average	St. Dev.	Avg. difference	Avg. error
Offense	25	15.44	4.94	2.23	6.37
Defense	40	14.95	4.57	1.55	5.38
QB	2	17.47	9.17	10.53	18.86
Off. w/o QB	23	15.27	4.57	1.51	5.29

The remainder of the first round:

Position	Number	Average	St. Dev.	Avg. difference	Avg. error
Offense	29	26.01	6.51	2.51	9.68
Defense	36	26.75	6.60	-0.34	7.17
QB	1	21.06	8.75	-4.06	4.06
Off. w/o QB	28	26.19	6.43	2.74	9.88

Other ranked players:

Position	Number	Average	St. Dev.	Avg. difference	Avg. error	% ranked
Offense	65	37.75	3.82	2.17	11.45	19.5%
Defense	58	37.55	4.06	7.77	14.30	22.0%
QB	9	36.75	5.79	-1.86	16.08	20.1%
Off. w/o QB	56	37.91	3.50	2.82	10.71	19.4%

Take-aways

• Defensive players ranked in the top ten are extremely accurately projected.
• Top ten ranked quarterbacks are likely "drop" candidates (particularly if they are not projected in the top three).
• Defensive players are much more likely to be projected in picks 10-20 than offensive players. However, the defensive players remain much more accurate projections than the offensive players.
• Defensive players projected to go in the last third of the first round generally go sooner than expected.
• Quarterbacks that appear in a few mocks, but not all, are good "reach" candidates.
• Defensive players that appear in only a few mocks seldom end up in the first round.

One final note

Since 2007, the following players were projected, on aggregate, to go higher than pick eleven, but lasted until the eleventh selection: Fletcher Cox, Nick Fairley, Robert Quinn, Bryan Bulaga, Dan Williams, Brian Orakpo, Jeremy Maclin, Branden Albert, Leodis McKelvin, Brady Quinn.