Starting Pitcher Report

[stembot]

I've tried using the Rating figure in this report in both my daily and season-long leagues. But when I see a guy like Joe Saunders assigned a +2.86, it really makes me question the value of the Rating.

I still find the report useful to simply know who is starting in an 8-day view. But other than that, I think this report needs to be upgraded.

Or, hey, someone illustrate how Saunders is as good of a start (vs PIT) as Samardzija vs HOU. Samardzija has to be substantially more valuable than Saunders. But they have nearly identical Ratings in today's report for this upcoming week.

[RAY@HQ]

The whole point of the report is that it's based on objective data, not your own subjective impressions. Saunders has a good rating this week for two reasons:

- he's been good at home.
- the Pirates have been terrible against LHP on the road. Note the oppScore of 4.00. That means that LHPs who face Pittsburgh away from Pittsburgh have averaged a PQS-4 this year. That's a pretty strong indicator.

Anything can happen in one game, and nobody is saying that Saunders is actually as good as Samardzjia for the balance of the season. But if the report is running contrary to your own gut, I think that's a good thing. It's certainly not an indicator that it needs to be "upgraded".

[stembot]

Understood on the objectiveness of it. I really have no opinion of Saunders other than what I see as his results and projection, and both are really bad. I'd like to think that's hardly an opinion/subjective.

I do understand he's got a great H/R split this season. Is 46 innings of each split this season a better indicator of his skills in the short-term, compared to his projection?

[Hardball006]

Every week there are a few guys like that. Once you learn how to use it, you''ll find it to be a priceless source packed full of data/info.

[Gantner]

Yep. Once you understand how it works (i.e., what gets plugged into it, as Ray mentions above) you also learn where to add your own input. My last pitching spot this week went down to Lackey at home vs. COL and Medlen on the road vs. KC. Lackey has a much higher score (3.16 vs. 1.08), but when I looked closer, I see that while Lackey has done really well at home this year, he only has four home starts making the sample pretty volatile (especially with three of those four starts vs. CLE, HOU, and MN). And while Medlen has been worse on the road than at home, that is largely because the bulk of his road games came in mid-May or earlier - which happened to be the part of the season when Medlen was putting up crappy BPIs both at home and on the road. He has only two road starts since then, which have been very good, and in the end I'm concluding that he has no real home/road split.

In the end, they're both good starts, but I see them as equal when you look at what went into the samples. I went with my gut and started Medlen whose last five PQS scores look better.

[Whistler]

Understood on the objectiveness of it. I really have no opinion of Saunders other than what I see as his results and projection, and both are really bad. I'd like to think that's hardly an opinion/subjective.

His last 5 starts arent that bad, averaging 7 IP and a little over 1 ER in each. PQS not great 4-5-3-4-3 I guess due to low K rates. 7 straight reductions in DIS% down to 33%.

Of course those trends are probably due for a correction.......

[stembot]

I have used the report (primarily in an attempt at some week-long leagues) and had limited success. Anecdotal, and maybe I don't have the magic eye that apparently some do with it, but it's my experience that using the report's Rating figure isn't a great idea.

That Medlen vs. Lackey example is great. I think the report needs to take into account that kind of thing, somehow. Of course, I have no solution...

Thanks for putting up with my complaints.

[stembot]

Is there data going backwards on these pitcher Ratings? How do they correlate to actual performance?

If Saunders were in the same situation again as he was last night (at home vs PIT) in a week, would you guys start him again?

[RAY@HQ]

Yes, this article is linked from the report: http://www.baseballhq.com/content/re...pitcher-report

[Gantner]

Just another thought on the report.

PQS has its place and purpose. It's shorthand, and an improvement over the standard QS definition. When people bring up its flaws it is rightly defended on HQ with terms like "quick and dirty," "simple," "at a glance," etc. It's not meant to be comprehensive; it's a way for us to glance at a box score and instantly evaluate a start without running a complex formula, while looking past some of its inherent flaws such as its dependence on H%, the fact that sometimes an outstanding CG start that "just misses" on a couple categories while dominating others registers a PQS-3 while a more ordinary 6-inning start that barely clears a few categories registers a PQS-4 or 5, etc.

I think what it boils down to is that PQS is made for humans. Computers are way overqualified for this; if they had feelings they would feel insulted to be put to the task of calculating a score made for humans to figure in a couple of seconds while they can do much more sophisticated computations in much less time. So why do we rely on PQS for the pitchers report?

I'm taking Travis Wood as an example. Here's a guy with a 57 BPV and 4.18 xERA on the year, in other words pretty much a dead-average pitcher. His average PQS is a whopping 4.2 however. He has lots and lots of starts where he just barely clears the criteria for a 4 or 5; on the other hand, yes, he is a very solid pitcher who seldom disappoints, and PQS admittedly does capture that fairly well. But I don't think it's good enough for the Starting Pitcher Report. I don't know his full BPIs in home games since HQ doesn't offer those splits, but I do know his home Cmd is 2.2 which is actually worse than his overall Cmd of 2.4. The Cardinals are one of the best offensive teams in all of baseball, not as good on the road but their 3.10 rating on the road vs. LHP is nothing to sneeze at. I'm not sure if I'd recommend Wood for his start tomorrow. I wouldn't rate it a negative, but subjectively I'm thinking somewhere in the 0.80-1.00 range, just looking at the opponent and the fact that Wood is seems to be the very definition of an average pitcher. But he gets a ringing endorsement of 2.28 on his report, all on the basis of PQS.

Of course my "0.8-1.00" guess is purely subjective and has very little science behind it. Maybe the report is right and I'm wrong. It just doesn't seem to pass the smell test here, and in this case it's not due to small sample size. I think the problem is PQS.

As for small sample sizes, it would be great to somehow use HQ's projections (which incorporates past performance) in place of just past performance YTD, since HQ's projections already solve the problem of sample size. Of course, you can't project PQS so if we're using PQS in the report, this idea is dead in the water. But if it was something other than PQS ... or if somehow we could devise an xPQS ...

Probably better for Think Tank. If people like this line of discussion I'll move it over there, but for now, the topic of discussion is established in this forum so I'll start here.

[Whistler]

As for small sample sizes, it would be great to somehow use HQ's projections (which incorporates past performance) in place of just past performance YTD, since HQ's projections already solve the problem of sample size. Of course, you can't project PQS so if we're using PQS in the report, this idea is dead in the water. But if it was something other than PQS ... or if somehow we could devise an xPQS ....

The problem with using ROS projections for a daily decision (then adjusted for home/road, strength of opponent, strength of opponent vs handedness, etc), is that HQ self-admittedly is slow (conservative) in adjusting ROS projections, mainly with regard to playing time. So you would have to use some sort of BPI index. But isnt that what BPV is?

[Gantner]

I don't think playing time would come into play here, but yes, something BPV-ish. Perhaps PQS could be more like a per-game BPV, using things like GB% and HH% to replace hits and HR in the PQS formula. I do like how PQS also gives us a sense of consistency by showing us DOM% and DIS%, although admittedly DOM% and DIS% only indirectly reflects in the Starting Pitcher Report and I see no reason to change that (a "volatility index" separate from the recommendation score might be helpful however).

If we had that sort of metric, we could perhaps project how a pitcher will perform in all home games for the rest of the season, for example, and express that in some sort of per-game basis.

[Gantner]

Also, while HQ is conservative in bumping playing time projections, I'm fine with how it projects skills in the vast majority of cases. At this point in the season it should be a fairly even mix of current performance and pre-season projection, hopefully leaned toward current performance assuming the player has played almost the entire first half of the season.

[RAY@HQ]

Just another thought on the report.

PQS has its place and purpose. It's shorthand, and an improvement over the standard QS definition. When people bring up its flaws it is rightly defended on HQ with terms like "quick and dirty," "simple," "at a glance," etc. It's not meant to be comprehensive; it's a way for us to glance at a box score and instantly evaluate a start without running a complex formula, while looking past some of its inherent flaws such as its dependence on H%, the fact that sometimes an outstanding CG start that "just misses" on a couple categories while dominating others registers a PQS-3 while a more ordinary 6-inning start that barely clears a few categories registers a PQS-4 or 5, etc.

I think what it boils down to is that PQS is made for humans. Computers are way overqualified for this; if they had feelings they would feel insulted to be put to the task of calculating a score made for humans to figure in a couple of seconds while they can do much more sophisticated computations in much less time. So why do we rely on PQS for the pitchers report?

I'm taking Travis Wood as an example. Here's a guy with a 57 BPV and 4.18 xERA on the year, in other words pretty much a dead-average pitcher. His average PQS is a whopping 4.2 however. He has lots and lots of starts where he just barely clears the criteria for a 4 or 5; on the other hand, yes, he is a very solid pitcher who seldom disappoints, and PQS admittedly does capture that fairly well. But I don't think it's good enough for the Starting Pitcher Report. I don't know his full BPIs in home games since HQ doesn't offer those splits, but I do know his home Cmd is 2.2 which is actually worse than his overall Cmd of 2.4. The Cardinals are one of the best offensive teams in all of baseball, not as good on the road but their 3.10 rating on the road vs. LHP is nothing to sneeze at. I'm not sure if I'd recommend Wood for his start tomorrow. I wouldn't rate it a negative, but subjectively I'm thinking somewhere in the 0.80-1.00 range, just looking at the opponent and the fact that Wood is seems to be the very definition of an average pitcher. But he gets a ringing endorsement of 2.28 on his report, all on the basis of PQS.

Of course my "0.8-1.00" guess is purely subjective and has very little science behind it. Maybe the report is right and I'm wrong. It just doesn't seem to pass the smell test here, and in this case it's not due to small sample size. I think the problem is PQS.

As for small sample sizes, it would be great to somehow use HQ's projections (which incorporates past performance) in place of just past performance YTD, since HQ's projections already solve the problem of sample size. Of course, you can't project PQS so if we're using PQS in the report, this idea is dead in the water. But if it was something other than PQS ... or if somehow we could devise an xPQS ...

Probably better for Think Tank. If people like this line of discussion I'll move it over there, but for now, the topic of discussion is established in this forum so I'll start here.

Good post, but I think you're underselling PQS. Yes, we designed it to be easily accessible, in a "quick and dirty" way. And when looking at individual starts, that's going to lead to the appearance that some guys are "beating the system" by just squeaking into a particular PQS score that it looks like they may not deserve (good or bad). But overall, as you start aggregating large numbers of starts, that all washes out in the larger sample sizes.

Wood's skills are average, yes. But his results are better than average, as evidenced by his 2.69 ERA. Now, we know that PQS doesn't look at earned runs at all. But while Wood has a 2.69 ERA, his 78%/0% PQS Dom/Dis split correlates to a 2.84 qERA (that is, the ERA that would be predicted from his PQS scores). That tells me that the "quick and dirty" PQS is doing a very nice job of capturing what Wood has done so far this year. The xERA is more dubious of his results to date, and that's a reason for caution. But just because two metrics are in conflict doesn't mean that's automatically an indictment of PQS.

Are there ways that we could have done this differently, maybe comparing the pitcher's xERA to the BPV of the opposing lineup, rather than using PQS? Sure. But I'm not sure they would be any better at this, in fact I don't think they would be. What we have right now isn't perfect, but it's pretty darn good. And when trying to put a prediction on the outcome of one game, I think that's the classic case where you want to focus on being good, and not letting the unwinnable chase for perfection get in the way of the good.

I don't want to keep just pointing to the research article linked on the report, but the data there does sort of speak for itself. Look at this table:

Rating     Count   Wins   Win %  Avg PQS   DOM%    DIS%
=======    =====   ====   =====  =======   ====   ====
  >3         95     47     49%     3.7      68%    12%
 2 to 3     939    375     40%     3.5      60%    13%
 1 to 2    2362    896     38%     3.2      51%    17%
 0 to 1    1674    554     33%     2.9      42%    22%
-1 to 0     462    140     30%     2.6      39%    29%
-2 to -1    268     81     30%     2.5      29%    27%
  < -2      118     23     19%     2.1      28%    41%

You can see here that the ratings correlate directly to better PQS scores. And we know from other research (as mentioned with Wood above) that PQS scores correlate very well with ERA. Therefore, these ratings correlate to ERA. I'm more than willing to dump the data every season to keep showing this, but I really think everyone keeps focusing on the negative examples and ignoring the positive ones. This thing doesn't work perfectly, but it works pretty darn well at quantifying a particularly difficult problem.

[Gantner]

Good explanation. Thanks Ray.