Fantasy Hockey Scouts: Projected Stats... Who's The Best?

(Actually the title should probably read "Projected Stats... Who Sucks The Least?"... but it doesn't have as nice a ring to it)

ProjectionsIn our pre-FHS days, we would wonder how accurate those projected stats (that we laid hard earned dollars down for) would end up being.  We would hear sites and magazines rattle on about how their projections were the best but we never bothered to check their claims... until now.

Using our biggest competitor's projections from September 2009 we've taken a fairly in-depth look at the numbers to see who performed the best and how accurate they were. 

We looked at four of the biggest fantasy sites out there (ummm, okay... well us and three of the biggest)... Rotowire, Dobber Hockey, CBS Sportsline & Fantasy Hockey Scouts.  Why didn't we include ESPN?  Well because they are unbelievably bad!  We sincerely hope that you didn't use ESPN's projections for your fantasy draft... we did warn you.

THE FINDINGS

CrosbyThe analysis shows us that projected stats aren't very accurate... at least they weren't last year and none of the sites can really claim to being that great at it:

- Number of players projected: FHS (533), RW(352), DOB (498), CBS (526)... we only compared the players (309) that all four had in common.

- On average the four fantasy sites were 13-15 points off the actual with their projections.  When you consider injuries, demotions, etc... this isn't a terrible differential but it ain't great. 

- Only 23%-30% of the projections were within 5 points of the actual... not numbers to be proud of.

- 40%-47% were within 10 points of the actual, 72%-78% were within 20 points and if you give a 30 point cushion either way then the sites will be right 91%-94% of the time.

- Patric Hornqvist (51), Tyler Myers (48), Michael Del Zotto (37), Manny Malhotra (33), Vern Fiddler (30) and Marcel Goc (30) where the top six point-getters that nobody predicted... guys like this are pretty rare.

- All four sites consistently over-project... particularly on guys with lower point totals (i.e. there's always a lot more guys that bomb then what is predicted).

As you can probably tell from the ranges above the four sites are very close together in terms of their success rates but from our analysis they rank as follows (see below for the detailed comparison):

1) Fantasy Hockey Scouts:  The numbers say we're #1 but it's close, particularly between first and second.  We had the lowest average error (13.52) and the highest % within 5 points of the actual (30%) for the 309 players and finished just a fraction below Dobber on the Top 100 scorers analysis. 

2) Dobber Hockey:  They had the lowest average error (11.18 vs our 11.30) and highest % within 5 points (34% vs our 32%) if you just look at the top 100 scorers but they were only third best when looking at all 309 players. 

3) CBS Sportsline:  They were second best with their projections of all 309 scorers with a 13.64 average error & 25% within 5 points of the actual but were a fairly distant third when looking at just the Top 100.

4) Rotowire:  Surprisingly, they had the most perfect predictions (11) but were the worst overall in terms of average error and % within 5 & 10 points for both Top 100 and all 309 players... their performance made it a pretty easy call for the bottom spot.


THE CONCLUSIONS

ToewsAs expected, no site has got it all figured out when it comes to projecting statistics.  I mean, when our 30% success rate (within 5 points) tops the group then you know that you can't really trust projections any more than you can trust your local weatherman.  With so many variables to predict, nobody can do it with a great amount of accuracy... don't believe anybody who says differently.

So does this mean projected stats are a waste of money?  We'd like to think otherwise and what you really need to remember is that, no matter whose projected stats you use, the rankings are important... not the numbers.  Don't get hung up if the projected stats are saying a player is expected to get five more points than the guy you really want... take the guy you want.  What it's really all about is picking guys with upside.  That's why we spend so much time putting documents like the Breakout Bible, Fantasy Scouting Report and Fantasy Prospects DB together for fantasy GMs.  We're much more proud of the fact that we were the site with the highest point projection for Steve Stamkos last year (even though we were still 19 points off) than we are about projecting Ryan Malone for exactly 47 points.

The difference between the four sites isn't substantial.  According to our analysis we were the best but when the projections are all over the place, a few points here and there won't amount to anything substantial.  So when you're picking your fantasy provider don't decide based on how accurate they claim their projections are... we would recommend that you factor in cost (we're the cheapest btw) and what other elements are in their draft kit.

Follow the jump for more detailed analysis including charts, graphs and some notes on our analysis.

THE COMPARISONS

We divided our comparison into three sections:
A - The 309 players that every site made a prediction for
B - Top 100 scorers in 2009-10
C - A closer look at the top 5 breakouts and the top 5 scorers

A - All 309 players

Difference of Projection from Actual
RW

DOB

CBS

FHS

Average Error 14.81 14.40 13.64 13.52
Standard Deviation 11.0 10.7 10.8 10.4
Median 13 13 11 11
Correlation Coefficient (1 is perfect) 0.64 0.74 0.71 0.73
Biggest Over
54 (Sykora)

56 (Savard)

58 (Savard)

55 (Savard)

Biggest Under

-44 (Doughty)

-33 (Stamkos)

-40 (Bergfors)

-35 (H.Sedin)
# of Players Overprojected 191 233 212 215
# of Players Picked Exactly 11 2 8 9
# of Players Underprojected 107 74 89 85
         
% within 5 points of actual 23% 24% 25% 30%
% within 10 points of actual 40% 45% 47% 46%
% within 20 points of actual 72% 74% 78% 74%
% within 30 points of actual 91% 91% 92% 94%

Like we said up top, FHS comes out #1... but it's pretty close.  We're followed by CBS, Dobber and Rotowire.

B - Just Top 100 scorers in 2009-10

Difference of Projection from Actual
RW

DOB

CBS

FHS

       
Average Error 14.80 11.18 11.80 11.30
Standard Deviation 11.0 9.0 8.7 8.7
Median 13 9 10 9
Correlation Coefficient (1 is perfect) 0.55 0.61 0.58 0.61
Biggest Over 37 (Spezza) 43 (Malkin) 34 (Malkin) 33 (Malkin)
Biggest Under -44 (Doughty) -33 (Stamkos) -36 (Stewart) -35 (H.Sedin)
# of Players Overprojected 35 56 48 48
# of Players Picked Exactly 3 2 1 2
# of Players Underprojected 62 42 51 50
   
% within 5 points of actual 23% 34% 29% 32%
% within 10 points of actual 44% 59% 52% 53%
% within 20 points of actual 76% 84% 84% 84%
% within 30 points of actual 89% 97% 96% 97%

Here's a another look at how the four websites went (click on the picture for a larger view).  This plot shows how we did relative to the actual point total.  Ideally, all of our data points would be right along the center line.  As you can see none of the four sites are close. 

You can also see the trend that as the actual points get larger (right side of plot), the closer our projections get.  As the point totals drop, so does the accuracy. 

Projected Stats 100 
Just looking at the Top 100, Dobber has the slight advantage over FHS, CBS is in third place and Rotowire brings up the rear again.


C - Finally, here is a a closer look at some specific players: Top 5 Breakouts and Top 5 Scorers

Top 5 Breakouts Actual

FHS

Other guys
Steven Stamkos 95 76 56, 62, 63
Chris Stewart 64 31 30, 38, 28
Dustin Penner 63 44 51, 43, 38
Drew Doughty 59 37 15, 38, 38
Rene Bourque 58 60 31, 43, 35
       
Top 5 Scorers Actual FHS Other guys
Henrik Sedin 112 77 71, 82, 79
Alex Ovechkin 109 115 112, 120, 111
Sidney Crosby 109 111 124, 113, 109
Nicklas Backstrom 101 86 80, 92, 87
Steven Stamkos 95 76 56, 62, 63

You think we could at least get the Top 5 Scorers correct?... but all four sites missed out on the superstar breakouts of Sedin and Stamkos.  As for the breakouts... we thought Stamkos & Doughty would be good, but not that good, so only FHS picked even one of the Top 5 breakouts accurately (Bourque).


THE NOTES:

Oshie- We only compared numbers when all four sites projected a total for that player... 309 players fit this criteria.

- We didn't adjust for injuries in our calculations (which is the biggest reason the projections are so inaccurate).  Many injuries could not be predicted but shouldn't a site that predicted a big season from Gaborik get credit for that?  We think so.

- Lists are done and tweaked constantly leading up to the start of the season.  These stats came from the respective sites at different periods in September.  There will be some bias in the analysis as some sites might have had a few extra weeks to get their numbers tweaked.  The FHS numbers were from the middle of our tweaking process (September 23rd).

- We are dealing with over 700 players.  Often sites miss a player that has no reason being left off the list (i.e. Rotowire didn't have a projection for Loui Eriksson in the stats file we used).  Mistakes happen when you deal with numbers this large.

- Some sites project shots, PPP, SHG, etc. while others only give you the basics like goals.  All four project points, so to make things fair for all we've just looked at the point projections. 

X
Log In Sign Up

forgot?
Log In Sign Up

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

Join Fantasy Hockey Scouts

You must be a member of Fantasy Hockey Scouts to participate.

We have our own Community Guidelines at Fantasy Hockey Scouts. You should read them.

Join Fantasy Hockey Scouts

You must be a member of Fantasy Hockey Scouts to participate.

We have our own Community Guidelines at Fantasy Hockey Scouts. You should read them.

Spinner

Authenticating

Great!

Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.

tracking_pixel_9355_tracker