The Margin of Error of Projections

[jackvdo]

Thanks and Analysis Methodology

Just wanted to say Thanks to UsusalSuspects for last year's BBHQ forecasts and a thought about what I said before about the different ways of doing an analysis. I originally started analyzing ERA projections using Correlation, a statistical method which measures the correlation between two sets of numbers, which I believe most other analysis I have seen used. Doing it this way, Baseball Information Systems projections did not fare well. I decided to do it another way, using Average Deviation, which measures the difference between the projected and actual peformance of each individual player, which in my mind is actually what we want to know, and BIS numbers came out very well, because they seemed to do very well at predicting those players that had a huge swing in their numbers. I just thought this was interesting and food for thought.

[PD@HQ]

Right-to be fair you'd want the last projections before Opening Day from all contestants.

[Michael@HQ]

Ouch!

[bankerboy]

One needs to use a cud-off date closer to your other predictions if one wants to compare them.

Wow, those are "chewy" projections. *groan*

[Michael@HQ]

The Forecaster predictions are made in November. One needs to use a cud-off date closer to your other predictions if one wants to compare them.

[Yardsmen GM]

You could also buy last years' Forecaster...

[usualsuspects]

I am going to have to do it without BaseballHQ's 2006 projections, which is disappointing.

I have HQ's final 2006 preseason projections file right here on my PC and would be happy to forward it to you if you'd like. I also have 2005 and 2004.

[gtgator]

I agree that, for me, HQ is there to help spot trends (up or down) in players as well as identifying those players whose stats are supported by the skills and those supported mainly with luck. And, as with bankerboy, I find it most helpful in the end game.

I also agree that picking one stat and comparing projections is inane. In fantasy, I only care about how close a projection is in the stats we use in our league. OPS may be great - but isn't one we use, so why do I care who did best at predicting that?

Even then, I understand the vagaries associated with wins and saves, which means those stats are essentially educated guesses. I mean, in 2005, if I told you Clemens would pitch all year and have a sub-2.00 ERA, I doubt anyone would have projected only 13 wins. And same goes for a hitter who has several balls hit off walls during the year as opposed to being hit a couple feet higher and having them become HR - literally a fraction of an inch difference in the point of contact between the bat and ball. No prediction system can predict that. RBIs, SB opportunities (and therefore SBs) are also contingent on external factors outside the player's control. As such, predicting actual numbers will always be tough.

But a prediction system that can predict skills that will be exhibited? That is where I think HQ is best (along with playing time). How those skills will turn into actual numbers is contingent on plenty of factors that can't be predicted, such as luck, personal issues, etc.

As for PECOTA - I think to some extent, they use skills as well, though they either don't know it or won't admit it. By comparing a current player to players with similar builds, ages and past performance of those players, they seem to guess that a player will do X -where "X" is the average of the comparable players (along with other factors such as home park, etc.). Since those averages are basically based on past performance, and since past performance is based, in part, on skills, they are effectively attributing the comparable players' skills to the current player. But, again, while I think it is similar in some aspects, I still feel HQ is better as it delves deeper into what the current player has done and shown as compared to examining what other players have done.

[jackvdo]

Just an addendum to what I said above as I looked at the blog referred to by Michael above. In my humble opinion, that analysis is flawed because it only looked at batters with 500+ PA and it didn't say what pitchers, but I assume those with a lot of IP. I think we all could reasonably forecast the performance of those players with that large of a data set. As I said, it is those players with the small set of stats where the pros really shine. My ERA analysis looked at all players and BP clearly outpaced BBTF.

Of course to be totally honest, this reinforces Mr. Shandler's perspective of the uselessness of such an analysis but again I would maintain the usefulness is in for what purpose the analysis is done.

[jackvdo]

Analysis Would be Good and Fun - Why Not?

I just joined BaseballHQ and came across this thread which is my main interest right now. The reason I joined was hoping to get last year's projections as I am doing an analysis right now on the various commercial projections and my own amateurish attempts from last year, starting with ERA.

After seeing Mr. Shandler's post above, I see that I am going to have to do it without BaseballHQ's 2006 projections, which is disappointing. While I understand Mr. Shandler's objections to such a thing, though not the forecefulness of his objections, if the purpose is only to determine which projectionist did the best last year because that is likely to change from year to year and would depend on the methodology of the analysis - you can make numbers say anything you want them to say if you try hard enough - there are other things to be learned. My purpose is to weed out the fakes from the professionals and to see if different pros were better in different areas.

What I have learned so far - my own ERA projections of last year based upon a 3 year weighted average of DIPS adjusted for park/league changes, age, etc. actually held up really well against the pros when I had enough data, no. of years in ML and IP. For those pitchers who didn't pitch the previous year or very little in the major leagues, this is where the pros really shine against an amateur like myself and in my opinion, why they are valuable, if indeed they are pros.

The most interesting thing I have learned so far though, is how totally useless a pitcher's previous years ERA was in projecting this year's ERA. My guess is a lot of people already know that but I was astounded as to how worthless it was, the absolute worst of about a dozen different ERA projections/actuals I tested. Goes to show how worthless ERA is as a stat in and of itself in evaluating the performance of a pitcher.

Now to the subject that I imagine Mr. Shandler wishes I do not address here, but I will. So far, I have only been able to obtain 2006 projections from Baseball Information Systems (Bill James), Baseball Prospectus and Baseball Think Factory. Baseball Prospectus was the clear winner, not even close, in particular their PERA (Peripheral ERA). While I suspect that Mr. Shandler's projections would compare favorably to those of Baseball Prospectus, I guess I will never know, or at least I won't know until next year.

[bankerboy]

Frankly, what HQ taught me is to buy skills and that skills lead to playing time. The breakout candidates become more obvious when viewed this way.

I agree with this 835.9%. I have a pretty good feel for what the Konerko's and Pujol's of the world are going to do. It's HQ's view at the underlying skills that points you towards the difference makers.

In a 16 team mixed league (with 25 rounds). HQ's tools and discussion boards led me to these as my last six picks.

20) Juan Rivera
21) Scott Linebrink
22) Jose Lopez
23) Josh Willingham
24) Justin Verlander
25) Fernando Rodney

Every provider will give you the same main course. It's the little things at the end, like seasoning and sauce that make the difference.

[maisany]

Why Should It Cost You?

To me, that 70% pronouncement really hits the nail on the head.

For those of us who play auction, where a dollar value has to be assigned to each and every player rostered, this means that even those players who fall within the 30% of correct stats projections will have their dollar values adjusted somewhat by the relative changes to the 70% who were wrong, so even though their projections were dead on, we will have paid an inappropriate amount for them based on the corrections required from the incorrect segment.

It's like what Bruce Lee says in "Enter The Dragon":

It is like a finger pointing away to the moon; focus on the finger and you will miss all that heavenly glory.

Use the force, Luke.

[JVR]

I don't know how this thread spiralled into a discussion of us versus them -- I am fully sold on HQ, and hence don't see the value of comparing how some other site moves its decimals around.

My only intent was to help me get a better gauge of HQ's methods within itself, so as to help me more effectively navigate across the overwhelming data HQ has to offer. Hence, my questions about stat reliability versus past years.

However, I have to admit that reading the gaming part of the Forecaster gave me the perspective I sought. I am still a novice at this, and it does take a lot of time for the concepts to fully sink in -- especially when it comes to viewing several factors together in order to fully appreciate prevalent trends.

R$ values are fine and dandy -- and we all use them to a certain extent to drive points across. But the other stuff, the determination and monitoring of skills -- that's just priceless, the more you delve.

[RAY@HQ]

Frankly, what HQ taught me is to buy skills and that skills lead to playing time. The breakout candidates become more obvious when viewed this way.

You have learned your lessons well, young Jedi warrior. :-)

In all seriousness, this is a good point that shouldn't be lost. I was just trying to point out that another shortcoming of OPS-based comparisons is that there's no playing time component. My 840 OPS over 400 AB is a different projection than your 840 OPS over 250 AB, or 550 AB.

[NICK@HQ]

Just to echo Ron's point, the other aspect of this where I think HQ outshines everyone else is in projecting playing time. Not to toot our own horn, but I think we put more effort into that side of the projections than anyone, and it shows in the output. Again, that's something that is lost in OPS comparisons, but is perhaps even more critical to a "good" projection.

Frankly, what HQ taught me is to buy skills and that skills lead to playing time. The breakout candidates become more obvious when viewed this way. I get what they are doing with PECOTA, but I've never fully warmed up to the concept since it's just a more thorough version of the method of looking at the past to guess the future trends. Yeah, that can work for the general case, but it tends to miss the outliers (who tend to be outliers for a variety of reasons that BPIs can sniff out).

If I want projections for the general case, any ol' book or method will get you the idea that Jeter will have a high batting average and score a lot of runs, and that Big Papi will hit a bunch of dingers. I don't care about those cases. I want to catch the up-and-coming guys who don't have enough of a track record for PECOTA to be as relevant, but from the moment they picked up a bat in low-A have been laying down BPIs to be deciphered to a fair degree of accuracy.

YMMV, but for me the methods HQ uses matches what feels right to my analytical mind. I don't care if a $24 projection winds up being $20 or $29. What I care about is making the 70% percentage play which, in the long run, will pay off whether you are playing baseball or cards at Vegas.