How goalies age, and the survivorship bias problem

Older goalies can still get the job done, but sometimes they need to sit down and rest for a bit.
Photo by rubyswoon via Flickr

This morning, Steve Burtch published a study arguing that goalies do not show any appreciable decline with age. The key plot was this one, in which the x-axis is the goalie’s age and the y-axis is how many standard deviations above or below league average he was (for goalies with at least 30 games played):

Goalie vs age

It’s clearly true that the observed performance of old goalies isn’t appreciably worse than the observed performance of young goalies. The problem is that word "observed" — we don’t get to observe the performance of all goalies at all ages. JaredL previously looked at this issue and showed how few goalies continue to play heavy minutes into their late 30’s. This creates what is called a survivorship bias.

A quick thought experiment

Let’s take Steve’s claim at face value, that goalies do not decline appreciably with age. Let’s further assume that the best goalies are the ones most likely to have long careers. Those top-tier goalies would have excellent save percentages in their late 20’s, and (per our first assumption that they don’t decline with age) they would continue to post strong numbers in their late 30’s.

As the lesser goalies’ careers come to an end (per our second assumption) and only the premier goalies are left playing, we would actually expect to see a rise in the average save percentage — the only goalies left at age 37 would be our future Hall-of-Famers still performing at a very high level. Of course, that’s not what we see; average save percentage is basically flat with age, as both Steve and Jared showed.

So that means one of our assumptions must be wrong. Either a goalie’s likelihood of having a long career is uncorrelated with how good he is or goalies do decline with age.

Looking for survivorship bias

Steve’s approach can suffer from survivorship bias because he allows the list of goalies being compared to change with age — instead of asking whether 34-year-old goalies are worse than younger goalies as a whole, he should be asking whether they are worse than they were earlier in their careers.

His evaluation of performance at age 35 (to pick a year at random) includes 29 goaltenders. Of those, five are already in the Hall of Fame, and some fraction of Brodeur/Joseph/Kolzig/Vokoun/Thomas/Hasek will eventually join them (along with, perhaps, Osgood). The group of 25-year-old goaltenders is nowhere near as strong — we don’t expect 30% of those guys to make the Hall of Fame. As a result, comparing the elite group’s performance to that much broader cross-section is misleading; if the premier goalies only posted run-of-the-mill numbers in their later years, then this clearly represents a decline in their performance.

To illustrate the effect this has on his approach, I have reproduced his chart, but with a few changes:

  1. I highlighted in red the goalies who went on to play 30+ games after the age of 35
  2. I eliminated all goalies who played 30+ games last year and were 35 or younger, since we don’t yet know how long their career will be or what color dot to give them

The resulting plot looks like this:

Survivorship bias

Here we can plainly see that the set of goalies who play past the age of 35 is clearly an above-average group (look at how many of the below-average dots in the 20’s are blue) and that those goalies are indeed declining with age (note the steep slope of the regression line for those goalies). It seems clear which of our two assumptions is wrong.

Unfortunately, what I have done here in sorting goalies into two groups is the kind of threshold-based analysis that I have previously noted can be risky. So as one final check, let’s perform the same analysis, but this time moving the threshold from 35 to 33 to make sure we don’t get a markedly different result. Here’s how that plot looks:

Survivorship bias 2

Lowering the cutoff brings in some lesser goalies, so we see more red dots below 0 early in their career and we see the trend line for the group dropping below average at an earlier age.

But the overall result is exactly the same: goalies do decline with age, and the goalies who have long careers are the ones who are better than average to begin with and so can withstand more decline and remain viable NHL options.

Obviously this data can’t say whether the decline is because the older goalies suffer more minor injuries or lose quickness or reflexes or heart or whatever. Apparently Steve’s goal is to somehow sort out what causes the decline, and perhaps he will find something interesting there; I can’t say until I see that data.

But I wholeheartedly reject his current conclusion that "the correlation between goaltender performance and age is nil. Goaltender performance may not be random, but it’s NOT related to the age of the goalie playing the position at the NHL level" — once you account for survivorship bias, it seems very clear that goalie performance does decline with age.

  • You nailed it.

    The position has also changed dramatically during Luongo’s career. His technique trails every new goaltender who enters the league and as he ages he will rely more on experience.

    This is not a blueprint for continued success as he moves forward.

  • Since the NHL began tracking SV% in 83-84, only 20 Goalies have played to age 37 after having started their career in their 20s. (Belfour, Hasek, Roloson, Burke, Joseph, Brodeur, Khabibulin, Hedberg, Vernon, Healy, Osgood, Lemelin, Kolzig, Vanbiesbrouck, Roy, Moog, Hrudey, Fuhr, Essensa, and Barasso).

    From the ages of 27-30, this group averaged ~.005 above league average SV% for their respective seasons.

    By age 35, they are at league average for their respective seasons.

    By age 37 the group is at .01 below league average (.015 below where they stood relative to their peers at age 30).

    The difference of .015 over a 60 game stint would be 27 goals, (assuming 30 Shots against/game).

    Accordingly, the attrition rate after age 37 is remarkable. 20 goalies played at 37, 12 played at age 38, 6 played at age 39, 4 played at age 40, 4 played at age 41 (Belfour, Hasek, Roloson, Joseph).

  • I’d like to work a way to get around that. A goaltender who has a bad year in his 20s is probably way more likely to return than a goaltender who has a bad year in his 30s.

    “What causes the decline” would be interesting to look at. I think it’s disinterest more than physical toll. We currently have the best goalies ever to play the game in their 20s. I’m interested to see how guys like Lundqvist or Luongo do play through their 30s because it should answer a lot of important questions.

    • One of the things we’re exploring in our work is whether home nation is a factor in attrition. i.e. do European goalies tend to quit earlier as a result of decreased comfort etc.

      Also trying to look into injury history, coaches, etc.

      The whole purpose of the paper is to look at the attrition rate for goalies as they age.

  • Good work Eric,

    Just curious, if you’ve run the numbers on the group that’s done at 33 to see what happens to their numbers as they age.

    I haven’t yet, and I’ll get to it tonight when I’m done night school.

  • If you are going to do this kind of study, wouldn’t it make way more sense to track goaltenders as individuals and see what their career arc is in terms of level of play vs NHL average?

    For this sort of thing you want to know how a goalies performance is affected by his own age, not the age of the good and bad goaltenders in the league.

  • The results show that goalies’ sv% declines with age, not that “goalie performance declines with age.” They are not necessarily the same thing. It could be due to selection bias early in their careers.

    Take two identical goalies. One has a couple of lucky seasons and puts up a sv% better han his real skill level, the other is unlucky and has sv% below his real skill level. Which one gets the contract, and along with it the opportunity to demonstrate his regression to the mean?

    For real world example, see Theodore, Jose, and some unknown goalie with equal talent who was a career AHLer.