(Actually the title should probably read "Projected Stats... Who Sucks The Least?"... but it doesn't have as nice a ring to it)
In our pre-FHS days, we would wonder how accurate those projected stats (that we laid hard earned dollars down for) would end up being. We would hear sites and magazines rattle on about how their projections were the best but we never bothered to check their claims... until now.
Using our biggest competitor's projections from September 2009 we've taken a fairly in-depth look at the numbers to see who performed the best and how accurate they were.
We looked at four of the biggest fantasy sites out there (ummm, okay... well us and three of the biggest)... Rotowire, Dobber Hockey, CBS Sportsline & Fantasy Hockey Scouts. Why didn't we include ESPN? Well because they are unbelievably bad! We sincerely hope that you didn't use ESPN's projections for your fantasy draft... we did warn you.
The analysis shows us that projected stats aren't very accurate... at least they weren't last year and none of the sites can really claim to being that great at it:
- Number of players projected: FHS (533), RW(352), DOB (498), CBS (526)... we only compared the players (309) that all four had in common.
- On average the four fantasy sites were 13-15 points off the actual with their projections. When you consider injuries, demotions, etc... this isn't a terrible differential but it ain't great.
- Only 23%-30% of the projections were within 5 points of the actual... not numbers to be proud of.
- 40%-47% were within 10 points of the actual, 72%-78% were within 20 points and if you give a 30 point cushion either way then the sites will be right 91%-94% of the time.
- Patric Hornqvist (51), Tyler Myers (48), Michael Del Zotto (37), Manny Malhotra (33), Vern Fiddler (30) and Marcel Goc (30) where the top six point-getters that nobody predicted... guys like this are pretty rare.
- All four sites consistently over-project... particularly on guys with lower point totals (i.e. there's always a lot more guys that bomb then what is predicted).
As you can probably tell from the ranges above the four sites are very close together in terms of their success rates but from our analysis they rank as follows (see below for the detailed comparison):
1) Fantasy Hockey Scouts: The numbers say we're #1 but it's close, particularly between first and second. We had the lowest average error (13.52) and the highest % within 5 points of the actual (30%) for the 309 players and finished just a fraction below Dobber on the Top 100 scorers analysis.
2) Dobber Hockey: They had the lowest average error (11.18 vs our 11.30) and highest % within 5 points (34% vs our 32%) if you just look at the top 100 scorers but they were only third best when looking at all 309 players.
3) CBS Sportsline: They were second best with their projections of all 309 scorers with a 13.64 average error & 25% within 5 points of the actual but were a fairly distant third when looking at just the Top 100.
4) Rotowire: Surprisingly, they had the most perfect predictions (11) but were the worst overall in terms of average error and % within 5 & 10 points for both Top 100 and all 309 players... their performance made it a pretty easy call for the bottom spot.
As expected, no site has got it all figured out when it comes to projecting statistics. I mean, when our 30% success rate (within 5 points) tops the group then you know that you can't really trust projections any more than you can trust your local weatherman. With so many variables to predict, nobody can do it with a great amount of accuracy... don't believe anybody who says differently.
So does this mean projected stats are a waste of money? We'd like to think otherwise and what you really need to remember is that, no matter whose projected stats you use, the rankings are important... not the numbers. Don't get hung up if the projected stats are saying a player is expected to get five more points than the guy you really want... take the guy you want. What it's really all about is picking guys with upside. That's why we spend so much time putting documents like the Breakout Bible, Fantasy Scouting Report and Fantasy Prospects DB together for fantasy GMs. We're much more proud of the fact that we were the site with the highest point projection for Steve Stamkos last year (even though we were still 19 points off) than we are about projecting Ryan Malone for exactly 47 points.
The difference between the four sites isn't substantial. According to our analysis we were the best but when the projections are all over the place, a few points here and there won't amount to anything substantial. So when you're picking your fantasy provider don't decide based on how accurate they claim their projections are... we would recommend that you factor in cost (we're the cheapest btw) and what other elements are in their draft kit.
Follow the jump for more detailed analysis including charts, graphs and some notes on our analysis.
We divided our comparison into three sections:
A - The 309 players that every site made a prediction for
B - Top 100 scorers in 2009-10
C - A closer look at the top 5 breakouts and the top 5 scorers
A - All 309 players
|Difference of Projection from Actual||
|Correlation Coefficient (1 is perfect)||0.64||0.74||0.71||0.73|
|# of Players Overprojected||191||233||212||215|
|# of Players Picked Exactly||11||2||8||9|
|# of Players Underprojected||107||74||89||85|
|% within 5 points of actual||23%||24%||25%||30%|
|% within 10 points of actual||40%||45%||47%||46%|
|% within 20 points of actual||72%||74%||78%||74%|
|% within 30 points of actual||91%||91%||92%||94%|
Like we said up top, FHS comes out #1... but it's pretty close. We're followed by CBS, Dobber and Rotowire.
B - Just Top 100 scorers in 2009-10
|Difference of Projection from Actual||
|Correlation Coefficient (1 is perfect)||0.55||0.61||0.58||0.61|
|Biggest Over||37 (Spezza)||43 (Malkin)||34 (Malkin)||33 (Malkin)|
|Biggest Under||-44 (Doughty)||-33 (Stamkos)||-36 (Stewart)||-35 (H.Sedin)|
|# of Players Overprojected||35||56||48||48|
|# of Players Picked Exactly||3||2||1||2|
|# of Players Underprojected||62||42||51||50|
|% within 5 points of actual||23%||34%||29%||32%|
|% within 10 points of actual||44%||59%||52%||53%|
|% within 20 points of actual||76%||84%||84%||84%|
|% within 30 points of actual||89%||97%||96%||97%|
Here's a another look at how the four websites went (click on the picture for a larger view). This plot shows how we did relative to the actual point total. Ideally, all of our data points would be right along the center line. As you can see none of the four sites are close.
Just looking at the Top 100, Dobber has the slight advantage over FHS, CBS is in third place and Rotowire brings up the rear again.
C - Finally, here is a a closer look at some specific players: Top 5 Breakouts and Top 5 Scorers
|Top 5 Breakouts||Actual||
|Steven Stamkos||95||76||56, 62, 63|
|Chris Stewart||64||31||30, 38, 28|
|Dustin Penner||63||44||51, 43, 38|
|Drew Doughty||59||37||15, 38, 38|
|Rene Bourque||58||60||31, 43, 35|
|Top 5 Scorers||Actual||FHS||Other guys|
|Henrik Sedin||112||77||71, 82, 79|
|Alex Ovechkin||109||115||112, 120, 111|
|Sidney Crosby||109||111||124, 113, 109|
|Nicklas Backstrom||101||86||80, 92, 87|
|Steven Stamkos||95||76||56, 62, 63|
You think we could at least get the Top 5 Scorers correct?... but all four sites missed out on the superstar breakouts of Sedin and Stamkos. As for the breakouts... we thought Stamkos & Doughty would be good, but not that good, so only FHS picked even one of the Top 5 breakouts accurately (Bourque).
- We only compared numbers when all four sites projected a total for that player... 309 players fit this criteria.
- We didn't adjust for injuries in our calculations (which is the biggest reason the projections are so inaccurate). Many injuries could not be predicted but shouldn't a site that predicted a big season from Gaborik get credit for that? We think so.
- Lists are done and tweaked constantly leading up to the start of the season. These stats came from the respective sites at different periods in September. There will be some bias in the analysis as some sites might have had a few extra weeks to get their numbers tweaked. The FHS numbers were from the middle of our tweaking process (September 23rd).
- We are dealing with over 700 players. Often sites miss a player that has no reason being left off the list (i.e. Rotowire didn't have a projection for Loui Eriksson in the stats file we used). Mistakes happen when you deal with numbers this large.
- Some sites project shots, PPP, SHG, etc. while others only give you the basics like goals. All four project points, so to make things fair for all we've just looked at the point projections.