The Margin of Error of Projections

[RAY@HQ]

Just to echo Ron's point, the other aspect of this where I think HQ outshines everyone else is in projecting playing time. Not to toot our own horn, but I think we put more effort into that side of the projections than anyone, and it shows in the output. Again, that's something that is lost in OPS comparisons, but is perhaps even more critical to a "good" projection.

[jbluestone]

well PECOTA seems interesting to me... but im not sure it can really tell the whole story. I believe pecota finds a group of players from the history of baseball that are comprable to said player and use that (plus some other factors) to project performance. To me their projections always seem a little negative, so it seems they wouldnt project too many breakout performances) but would get collapse performances more often.

Honestly, its not the projections that i come to HQ for (though they are fun), but the analysis, and thought provoking ideas. In general any article i read on baseball tends to be about analysis or a new study, or trying to put value on something previously difficult to valueate. Like baserunning, fielding, etc. Im not concerned with whether reyes OPS was projected within .10 points etc (although it makes for fun debate).

[RON@HQ]

I was just invited again to participate in another comparative analysis, and I declined. When the inviter became more insistent about my participation, I shot back with this single response:

"With a stat like OPS being the most popular comparison gauge, I could have projected Jose Reyes' 2006 stat line, ended up with Richie Sexson, and be deemed a success (both OPS ~ .840). I have no interest in these types of exercises. Thanks."

Of course, BP will be trumpeting that PECOTA beat Shandler, which Peter Gammons will pick up on, and then all the world will see us as also-rans. If you want to judge my success based on how well I can project a Jose Reyes and end up with Richie Sexson, be my guest.

[JVR]

Got what I was looking for, straight from the Forecaster, page 47:

"Fact: The most advanced prognosticating systems are only going to be 'on target' 70% of the time. That leaves a huge variability in player value."

Now, this depicts the built-in discrepancy of projected stats as a function of player value, rather than statistical data per se. But it does provide perspective.

[jbluestone]

You know what would be interesting..... if every day we named an AL and NL hitter and pitcher.. and everyone on the board wrote down their projections... if we tally it up and find the mean... i wonder how accurate taht will be.... this is piggy backing off my post about request for greater participation in polls... what do u think?

[Michael@HQ]

I can't find the link but some blog had the numbers of accuracy and HQ was 1st by a decent amount.

I recall a blog where PECOTA was regarded as the most accurate set of predictions for 2006. Ron Shandler's BaseballHQ might have been in first place before the guy added in PECOTA. Hat tip to BaseballThinkFactory.org for pointing it out.

In general, I think Ron's position has been that he doesn't like these after-the-fact judging because the criteria ought to be more like which tout most positively impacts its customers ability to win, not a narrow mathematical question of who has the lowest variance from the final numbers (even if we could agree on how to do that precisely).

I may eventually get back to trying to evaluate tout projections myself. I collected the data a couple years ago, but it became too big a task for me. My sense is that there isn't much persistant difference in accuracy among the top tier of projections.

[JVR]

Thanks. I don't doubt HQ's would be amongst the very best, if not THE most reliable versus what's out there.

I was looking to peel the onion and see where within HQ's layers of stats, lay the greatest sources of reliability and error, from year to year.

[adichiara]

I can't find the link but some blog had the numbers of accuracy and HQ was 1st by a decent amount.

[JVR]

The comments on the HQ Players Statistics and Projections page provide helpful information on how the projections were put together.

However, what about the reliability of the projections as each baseball season comes to a close? Do we have a statistical measure of say, how accurate the 2006 HQ projections were, overall? How far off were they when the books closed on the final 2006 stats?

And what were the greatest sources of discrepancy outside of playing time issues (which I assume are probably the most unpredictable)?

By the same token, which stats proved to be the most reliable, by order of reliability?

Were the same sources of discrepancy present in each of the previous years or have these varied from year to year? For instance, were the greatest discrepancies in terms of counting stats or percentage stats overall? In the counting stats, which were the greatest sources of variance for both pitchers and hitters? How about in the percentage stats?

Just to be clear, I am not looking at leading indicators or any of the hidden stats this site uses to arrive at its projections -- just looking at the actual stats themselves that are relied upon in most leagues as scoring categories.

Thanking you in advance.