The perils of evaluating a hockey player based on a small sample

I don’t want to write any “I told you so” columns. Those can wait until March. I am cool with watching sportswriters twist in the wind without actually twisting in the wind.

From an October column, written when the Toronto Maple Leafs were 5-1 (and published after the Leafs went to 6-1) titled with the harbinger headline “Where are the critics of Nonis’ off-season moves now?” the Toronto Sun’s resident straw man assured Leafs Nation that Toronto’s controversial offseason moves made the team better:

“[The online statistical geniuses] yelled that Nonis doesn’t know what he’s doing and that tone continued throughout the summer after the deal for David Bolland, the buyout of Mikhail Grabovski, the signings of David Clarkson and Tyler Bozak and the awkward contract negotiations of Nazem Kadri and Cody Franson.”

“[They] make their hockey assessments via charts and graphs,” whined Simmons. “[they] all but murdered Dave Nonis when he traded for Jonathan Bernier.”

It’s an interesting contrast when presented with Simmons’ column from January 11. Since the 6-1 start, Toronto have gone 15-19-5 and, after Saturday’s action, dipped to 12th place in the Eastern Conference. Over the last few days, Carolina, Washington, the New York Rangers, Ottawa and New Jersey have all climbed above the Maple Leafs in the standings, and Columbus could theoretically do it Monday. They’re a point back with a game in hand.

Simmons’ assessment?

“Randy Carlyle was a success last season and thus far a failure this season. He is the same coach: His team clearly isn’t the same.”

Well… damn it all.

Not to rub salt in any wounds, but one of my major gripes with sportswriters is how quickly the majority leap on small sample sizes. Simmons and Brian Burke feuded, and you have to think Simmons loved every second of the Leafs having winning success a season ago after he got let go. I’m fine debating Simmons any day about why the Leafs had success last year: anybody who read this site will know that I believed that the Leafs winning was not due to toughness or grit or fighting or whatever it was, but a ridiculously high shooting percentage and an above-average save percentage, giving the Leafs the highest single-season PDO.

PDO, explained here, isn’t all that complicated. It’s incredibly volatile in a small sample, which is why I’m wary counting out the Leafs now. Goaltending and shooting are wicked unpredictable, and the Leafs could go anywhere over the next 30 games or so. If their goaltending and shooting are league average, however, they’ll continue to lose games, and that’s mostly because during the offseason, the Leafs prized the wrong talents. This was my assertion, and I’ll continue to stick to it.

I mention small samples. Remember, Simmons wrote this column when Jonathan Bernier had a .974 save percentage and a 0.84 goals against average. Since, Bernier has a .919 save percentage and a 2.83 goals against average. Again, not bad, but we’re no longer talking about a deal that will “surpass” the Roberto Luongo trade made by Nonis in Vancouver as a “piece of hockey thievery”, and it especially won’t considering Ben Scrivens is third among qualified goalies in save percentage (.931) and fourth in goals against average (1.97).

Rather than drop “I told you so” rants after a small sample of games, what I’m going to do is continue to disagree with last season’s narrative fallacies (it’s easier to prove something doesn’t exist than prove it exists, in the case of the Leafs toughness, all you need is a stretch like the Leafs since the start of the Leafs season after Simmons published his original column) and try to ignore the aspects of the game we can’t predict. It’s tough to build a team focused on intangibles because those talents are quite literally intangible, and you can’t tell if current success is going to lead to future success.

Jim Lang brought up a point from Twitter when talking about toughness:

There once was a time when Colton Orr served a purpose on the Leafs. Not anymore. As James Mirtle of the Globe and Mail pointed out on Twitter the other night opposing teams are just not engaging Orr and the Leafs in as many fisticuffs as they were last year. The Leafs will be better served having a fourth line that actually can play 10 effective minutes every night than what we are seeing right now.

I think a lot of sports media are too concerned with “what has happened” rather than “what will conceivably happen”. Seasons are long, and we have to take into account adjustments, regressions, injuries, or any unpredictable events that shake up the makeup of a team. Rookies play their way on. Veterans play their way off. They’re traded. Opposing coaches adapt to a powerplay strategy. The goalie develops a bad case of stomach flu and stops 80% of pucks for a two-week period…

That can be traced to what the Leafs were apparently doing a year ago that just doesn’t work this year. Teams have learned to play against the Leafs. They don’t have to engage them physically, and the Leafs’ best players aren’t necessarily physical. That’s not a bad thing, but the Leafs are trying to win with an unsustainable style. Maybe there are situations where a fight or a hit can rally a team to win. All I know is that the data tells me you can’t count on fights, hits, grit, or even talent, alone.

Hockey is a weird game, in that luck dominates a small sample and can lead people to curious conclusions. I don’t pretend to know the answers, but I do know what the answers aren’t. You can’t focus on one small aspect of the game and expect that this is the indicator a team is primed for success or failure, without doing the necessary legwork to see if a trend that presents itself in October will last in March.

The most important goal in hockey is “the next one”, which is what any team is playing for, whether the team is 0-0 or 6-1. Shifting your understanding of the game to tell me what “will” happen and “why” is probably what’s going to get me to pay attention again to mainstream sportswriters who love to fabricate a compelling story based on characters they love or hate and project them on the page.

  • Jeremy Ian

    Good column but there is definitely something wrong in the Leafs’ dressing room. If the season goes down the tubes, there’ll be a new sheriff in town, maybe two -and Leaf fans will experience deja vu all over again. Could we see some signs of life with these guys please???

  • Jeremy Ian

    It’s bizarre to me that some people either don’t understand or don’t care about small sample sizes. Did Simmons actually think Bernier was going to continue to play at a .974/.084 rate? The Leafs would have finished the season with 70+ wins if that was the case.

    Also, “it’s easier to prove something doesn’t exist than prove it exists” – I’m pretty sure you’ve got that backwards.

    • Nah, that’s a concept I’ve been working with over the last few months, found in a Nassim Taleb book repeating a Karl Popper concept. One observation alone can disprove a rock-solid theory, while no single observation can prove something to be correct.

  • Jrose11

    I agree fully that the issues that are plaguing the Leafs were present last year. I think you (the author) hit the nail on the luck when you credited luck for why the Leafs finished where they did.

    Here is what I think the Leafs problem is, and anyone reading this can verify this tonight or any other game (until they fire Carlyle and/or Nonis).

    Overall it does come down to luck but I think I know why that is.

    You know the Rolston Centipede? If not, look it up it’s hilarious. Well we’ve got the Carlyle Clump, and it’s the dumbest strategy in all of hockey. Does this sound familiar to you? One Leaf pressures the puck while the other four collapse to right in front of the net to try and block a shot. Now why is the Carlyle Clump a terrible strategy? Let’s examine.

    1) Blocked shots are lucky- You have no idea where the puck goes after it hits you. The goal is to stop it from getting to the net, but when blocking in the Carlyle Clump all this does is delay the inevitable.

    2) After the shot is blocked, think of where the Leafs are. All in a clump right in front of Bern…. I mean the goalie. Now the opposing team can easily use this to their advantage (and oftentimes does) by spreading out and covering the majority of the ice. Thus any puck that goes to these non-right-in-front-of-the-net areas usually goes to the opponent. Hence the stupid “losing the puck battles” narrative.

    3) Now if the opposing team is smart they don’t even shoot into the clump they just pass around, and since the clump is usually stationary they can move them out of position and score.

    4) And this is the most important because even if the team doesn’t score this is exactly what happens. The Leafs can’t get out of their own zone. Maybe they never give up a real scoring chance but they can’t attack when everyone is behind the face-off dot. They don’t get the puck back easily and when they do players are either tired and/or unable to get into a breakout formation quick enough.

    So (and sorry for this ridiculously lost post) this is one major reasons behind the Leafs struggle. They rely on lucky shot blocking to prevent chances, don’t get the puck back, and can’t counter attack. All this can be traced part in some way to the Carlyle clump.

    Are the Leafs a good team? I actually think they are, but at this point I’m not even sure. The fact is the Leafs need to be more opportunistic (since now we have not one but two decent goalies), and use their great offence to win games, instead of relying on this insane “defensive system” that will not work.

    In short Carlyle has got to go, before they trade away great pieces like Gardiner and Kadri for Steve Ott (made that up, but can’t you see that). The Clump sucks and I don’t care how good Mark Fraser is at it, it’s a terrible strategy. Alright I’m done.

    • Jeremy Ian

      The Leafs play this year is quite different than last. The Leafs have looked a lot worse than last year and it was easy to predictions their early season success wouldn’t last. Unlike last year where many of the Stats crowd predicted the Leafs demise but it never really materialized and they took the cup finalists to the brink. Those who watched the games could see the Leafs playing well not matter what the shot attempts count said, and they can see the Leafs playing worse and differently for whatever reason this year.

  • Jrose11

    I told you so!!

    The season is 45 games. That is a pretty small sample size to dismiss Clarkson who is well below his past historic average for SH%. He is not a player who will score 5 goals or whatever his rate is.

    And you have an even smaller sample size for the injured Bolland but I suppose you can make a conclusion that suits.

    And this season has been a sample of 45 games and you are claiming that is “more” significant than a sample size of 48 games last year when the team made the playoffs. Do you see not see the problem???

    The leafs are not he only individuals ignoring the the advice of small sizes and making conclusions based on them.

  • Jeremy Ian

    You will “ignore the aspects of the game we can’t predict. It’s tough to build a team focused on intangibles because those talents are quite literally intangible, and you can’t tell if current success is going to lead to future success.”

    I am on your page. This is a teleology. Teleologies are bad explanations because they are circular. We’ve known since Francis Bacon that scientists avoid these kinds of formula because they ultimately point to variables beyond human abilities to judge with certainty. Blah blah. Whatever. The point is, you have many centuries of philosophy on your side. It’s also why sportswriters who love character-driven narratives can contradict themselves shamelessly.

    “Regression the mean” is important because it tells us what the slope will look like adjusting for chance and randomness that loom large in small samples and bad story-telling. (True as much of the “over” performance last year as the ugly spectacle of “under” performance since Halloween)

    The Leafs have been so volatile it’s hard to know where the mean lies (Edit: the team’s relationship to the mean). Around what line do “random walks” stagger? Who knows with all the personnel changes. (Leaving aside all the more predictable effects of coaches who have figured out the Carlyle method, injuries etc) This is why intangibles keep getting dredged up from the sludge.

    I don’t want this to degenerate into a seminar. Leaf management banked on outlying results and are desperately hanging on to a model that yields diminishing returns to a mean we can’t really figure out anyway (Edit: team relationship to the mean). This is basically your argument, and I agree.

    But what do you do with so much uncertainty, when we can’t even be sure we know if the current underperformance is a property of the team (as the doomsayers are arguing) or a combination of random effects? I’m hearing two rival choruses.

    1. Big changes for an entirely different, hopefully ideal, track? (This is the doomsayers position).

    2. Little changes to nudge things onto a better slope? (A fair number of your readers here)

    Do you change the direction of the team slope, or do you move the slope altogether?

    So, Cam: Which would you choose? This may haul you out on the carpet to be judged by your peers (maybe Simmons reads your blog with knives sharpened). Give it a shot.

    • I think No. 2. I think only little changes are required right now. There are still some damn good hockey players on the team and I can see a Stanley Cup team being centered around Phil Kessel, James van Riemsdyk, Jake Gardiner and Nazem Kadri, along with however valuable Dion Phaneuf is and what Joffrey Lupul has left of his prime.

      I don’t KNOW what I’d do, personnel-wise, but the current setup isn’t working. I’d like to see the same roster with a standard defensive zone setup and less extreme of a focus on matching up.

      We’ll see what happens.

      • Jeremy Ian

        I’m inclined to agree. That’s what’s frustrating. It would not take that much to tweak here and there — it’s sort of within grasp.

        So much easier that a wholesale revamp; what manager has the appetite for that.

        You’d think, faced with the unappetizing, they’d go for the nudging option.

  • personally i think steve simmons likes to hear steve simmons talk about how great and wonderful steve simmons is and that steve simmons is the centre of the universe when talking all things sports. afterall, steve simmons is by steve simmons account the greatest wealth of knowledge. all one has to do is ask.

  • Quasijr

    The problems I have with this team rant that they are losing it has been a useless 4th line & the teams allround compete level.
    We don’t need Colton Orr or Maclaren, we need 4th line that can skate like the wind has adequate NHL ability to score, fore check & back check.
    We should never start sweating if this line ends up in our zone doing a defensive zone start against the best lines in the league.
    I also find the compete level of this team is non existent, hardly any fore check & a crappy back check, which means our defence is hung out to dry.
    Though Kessel has gotten better at back checking he needs to bring up that compete level on his forecheck & back check. Im not talking about trying to pound a guy into the boards I mean proper body position & leaning into your man. The same thing Gardiner has to do. No I don’t want Gards to play like Scott Stevens, I want him to play like Brian Leetch, positionally sound & just leaned & guided his man away from the net or made it hard for him to get to net & get shot away.
    Luck is what you make it, you have to be good to be lucky & lucky to be good. Which means you make your own luck by doing all the little things right especially when your a mediocre team like the Leafs. This team doesn’t or won’t do the things to win those games because they don’t create they’re “luck”. You can blame management & coaching but I think its time to have a good look at the core players & their deficiencies
    Yes I believe Carlyle is just a average to below average coach like almost 80% of the coaches in this league.