The fatal flaw of analytics

Whenever a hockey player doesn’t talk to the media, a common refrain from reporters is that “the media is how fans connect to players”.

That is true, but not in the way the reporters think. The reality is that the media that connects us to hockey players is the television and radio—we’re dependent on those mediums to glean as much information as we can get about a hockey game without ever being there. Television is the best, because it’s visual. Radio is second best, because you can follow the progression of the game in real-time. There’s a big drop-off from there. I dislike following a game via a boxscore because there’s no drama involved, and even worse is reading the game recap the next day in the local newspaper, because even a 700-word recap can gleam over 95 per cent of the game as if it never happened.

It’s not so much a player’s post-game quotes that fans want to be privy to, but to actions. We are just months removed from two of my favourite moments as a sports fan: Marie-Philip Poulin’s heroics in the gold medal game of the women’s Olympic hockey tournament, and Percy Harvin’s kick return for a touchdown in the Superbowl. I consumed both events on a television, have a vivid recollection of my mental state during those events, but have, curiously, no recollection of what either athlete said after their respective games. It’s funny how that works. Two weeks before the Superbowl, I was lucky enough to be in Seattle for the NFC Championship game, a game best-remembered for Richard Sherman’s post-game interview. To the 60 thousand of us in attendance, however, or celebrating in Seattle after the win, we were too busy partying to have any care as to what Sherman said. The event did not shape any aspect of the way the game was played on the field, but reporters made so much of it—reporters who bank on interesting post-game quotes to write their stories.

I don’t mean this post to read like a study in media criticism, but there seems to be a concern in Toronto that Brendan Shanahan won’t use the Toronto Maple Leafs’ analytics budget the way we’d hope for it to be spent.

Per Mirtle:

[The analytics budget] is something that I’m going to use. I do have some thoughts and I have some meetings with some people about that. But to the extent and how we’re going to use it – “Some of the clubs that I know that do use it – every club uses it for a certain purpose or has a certain statistic that they value. So it’s not just like ‘we’ll make all of the decisions in our organization based on Corsi or Fenwick or the entire sum of it.’ But there are pieces of it that they may value that they use to support some of their decisions.
“I think some of the people that I’ve talked to said they still feel that it’s something that you can use in conjunction with other information, not by themselves. I understand the concern and some of the trends that I’ve seen and read up on.

In a previous post, I made the point that it didn’t matter whether an educated fan consulted ExtraSkater or watched games. As long as he had an opportunity to consume the actions of players, he’s going to be no more wrong than a guy like Shanahan or Dave Nonis when it comes to projecting performances. That’s not meant to be a slight on Shanahan or Nonis, but more a testament to how the technology has evolved to allow the die-hard fans to watch any game they choose. You can work hard enough and have the most-informed opinion on any player in the world.

There’s a misconception that analytics ends with Corsi and Fenwick. Those two numbers, which indicate puck possession, don’t tell you much about particular players, but can be a starting point for gathering more information.

Over the last two seasons, Tyler Bozak has a Corsi percentage of 27.5% when he’s not on the ice with Phil Kessel. The end point isn’t “Bozak is terrible without Kessel” (although it would be a pretty good guess) but if you want to think analytically, you’d want to consult the video or break down the data even further to find out why the Leafs get out-shot 72-28 when Bozak is on the ice without his regular winger. Is he earning more defensive zone starts? Is he being asked to dump the puck in more? Is he caught in the offensive zone because he’s waiting for passes that are unfortunately not coming from players named Phil Kessel?

There’s a lot to gather in, and a lot of things one could research that don’t end with Corsi or Fenwick. Ideally, you’d break down situations and find out what lineup decisions, strategies and player combinations are the most successful, and then further break down why. To what extent does chance play a factor? To what extent does the coach’s breakout tactics play a factor?

You can figure out all those things, but ultimately it doesn’t matter if you don’t have enough resolve or conviction to implement a strategy based on the information you’ve discovered. In Mirtle’s same interview with Shanahan, it seems clear that he agrees Randy Carlyle’s defensive zone strategy did not work a year ago and that the Leafs were too late to implement changes.

It’s fair enough for Shanahan to say “we think Randy is a good coach and he’s committed to being less aggressive in the defensive zone, more aggressive in the neutral zone and more lenient with our talented defencemen”.

Shanahan didn’t say that, though. He said this:

“Our hope is that the success that he has had as a coach, he’s won a Stanley Cup, you look around at the different people are available, but Dave and I made the decision…”

And it reeks of yet another executive valuing the results over the process.

an interesting statisticCarlyle has been a coach in the National Hockey League for nine seasons. He is 1/2 in “winning Stanley Cups” when he has gone into a season without Stanley Cup-winning experience, and yet is somehow 0/7 as a winning coach. A more damning statistic appears to the left. It seems that since winning a Stanley Cup, Carlyle no longer has the ability to win a Stanley Cup.

But the lone 2007 Cup is the crutch and the excuse. The Leafs could spend a billion dollars trying to find out why the team gets outshot 72-28 when Bozak is on the ice but not Kessel, but that money is better off not being spent if nothing gets implemented to change the fact. Fans that are banking on the Leafs spending their analytics budget are really banking on the Leafs making tough decisions, such as not retaining Randy Carlyle, an easily-marketable coach because he has Stanley Cup-winning experience.

The critical difference between fans and management is that fans have no obligation to put the needs of themselves and their job security and networks over the actual needs of the organization. In hockey, the research generally indicates that the more aggressive plays with higher reward, such as skating the puck in or out of the zone rather than dumping it and chasing after it. That’s not dissimilar from other sports like football or baseball, when the math overwhelmingly shows teams punt too much (giving up possession) or bunt too much (risking giving up a precious out for a single base). In tennis, the research shows that players are too conservative on their second service, for fear of a double fault (even though a weaker second serve generally means a much easier return shot for the opponent and a huge advantage for the non-serving player).

Ultimately, a risky decision that fails will be judged more harshly than a conservative decision that fails, since the riskier decision, especially if analytics are involved, doesn’t have results or an easy narrative to back up on. I’m not certain if the Maple Leafs fight at a disadvantage with this kind of stuff, but it’s been clear over the last few seasons that management is willing to make moves that make the team look better in the short-term and ignoring longer term consequences.

Whether Shanahan spends the analytics budget or not doesn’t change that fact. The Maple Leafs still need somebody in place with the trust of ownership to make the tough decisions even if a section of the fan base disagrees. Every major Maple Leafs move since the James van Riemsdyk trade has been predictable and none have fared particularly well (check the comments on this post, compared to the present day beliefs in Toronto about Joffrey Lupul). I don’t expect the Carlyle extension to work out long term, but right now the excuses for keeping Carlyle were easier to make than the excuses for firing him, and that’s all that matters to the front office at this point.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  • not_SorjeBalming

    Good points Cam,but I would also consider something that Shanny said about the fact they wanted the players to know they had to work themselves out of this mental drama of collapsing at critical times. Shanny used the term “shuffling deck chairs” by firing a head coach and letting the players off the hook for their performance.

    While I agree RC is not a very good coach, I also think Shanny might be smarter than we give him credit for by NOT letting the players off the hook and wanting to see how they react coming back with RC.

    In my opinion this is a sort of short term test, for both the returning players and the HC. He wants to see who really is prepared to work hard on his game to turn this around. He also wants to see if RC can adapt as well and not use the swarm defense and archaic breakouts.

    Maybe he has someone in mind to coach the leafs at mid season (or earlier) and knows he can hire them at a drop of the hat. I do like the fact he is THINKING this whole thing through instead of just appeasing us fans.

    In the end…we are in this situation because of Brian “Born impatient” Burke to some extent.
    Maybe having someone who thinks long term is what we really need.

    • Burke is an interesting case study. On one hand he fails at basic talent recognition. On the other hand, though, he, more than any other general manager, grasped the risk of front-loaded deals and avoided making any in Toronto. The teams that signed those long deals are now beginning to pay the price, whereas I don’t think Burke signed anybody for more than five years.

      His best trade—acquiring Phil Kessel—was very much a trade with the long game in mind. By the end of Kessel’s first season it was commonly accepted the Leafs took a bath with the Kessel trade, and four years later it’s looking the other way. The Leafs clearly won that deal. That was a pretty bold move at the time, and one that opened Burke up to a lot of scrutiny and one he’d never be able to shake if it hadn’t worked out as well as it has.

      • There was more to Brian Burke’s hockey acumen then you have mentioned. When he came here this organization had been bled dry of good young talent by both Fletcher and Ferguson. It was Burke that picked up Bozak, Scrivens and Gustafssen by beating out other teams to them, they were not drafted, He also got Gardiner and Lupul for Beauchamin and we will not go into the fact that he got rid of the Toskella and Blake. He did rebuild the Toronto farm system. Just sayin.(spelling on names above may not be correct)

        This article points out some very interesting flaws of just following stats. as well as giving every fan credit that has an opinion as opposed to slagging everyone that has an opposing opinion.

  • Jeremy Ian

    Good way to summarize the aversion to thinking differently about evidence before making decisions:

    “Ultimately, a risky decision that fails will be judged more harshly than a conservative decision that fails, since the riskier decision, especially if analytics are involved, doesn’t have results or an easy narrative to back up on.”

    I read you making two claims:

    1. Small “c” conservativism behind the bench represents a determination not to revise one’s strategy in the face of new evidence. Good organizations finds ways to build in the right balance of “adversarial collaboration” to check the propensity to over-commit to a losing strategy.

    Maybe this is what the advent of Shanahan means; that Nonis-Carlyle don’t stick to what they think they know at the expense of what they could know. I hope so. But you seem skeptical — do I have you right?

    2. There’s a more intrinsic flaw in the TYPE of evidence generated by analytics, that it’s hard to add up to a compelling narrative with all the right action verbs (fight, compete — which has been repurposed into a noun — etc).

    Good managers need to over-ride the tendency to favour one kind of evidence of the other.

    What this all means for Shanahan-Nonis-Carlyle is unclear. I don’t think the extended contract with Carlyle points one way or the other. Your Mirtle citations suggest a check on Bias 1. Can the group counter Bias 2?

    We do have evidence that Shanahan is aware of the dangers of fixating on any single narrative. In The Star, this is how he’s quoted:

    “When I was in Detroit, they were able to point back at being swept by New Jersey in the finals as an important part of their development rather than this ugly stain they couldn’t get over,” said Shanahan. “They can tell those stories because they didn’t quit.

    “Those failures became part of the narrative of their victories.”

    Maybe he doesn’t have a PhD in literary theory, but Shanahan is smart. Narratives are created after the fact. I tend to agree with the people who think this is a stop-gap measure; the contract extension is a way to signal to RC that he has no excuses and for Shanahan to have a Plan B. Given all that went on this spring (including getting a new job), Shanahan may not have felt able to get a firm grip on a complicated situation.

    • Good points here. The best indicator would be how the team handles the Dave Bolland situation this summer. A year ago the team spun this narrative that they wouldn’t have lost Game 7 had Bolland been on the ice and not Mikhail Grabovski, and I don’t think that’s true at all.

      We know that the group of Nonis, Poulin and Loiselle leans one way in regard to heart and compete level. Does Shanahan have enough pull to bring them back to normalcy?

      • rw970

        I don’t know but I would assume that former players, even former star players, are fond of the gritheart narrative if only because it provides them with a moral reason for their own success. No one wants to believe they’re just intrinsically better – they want to believe their success is earned. Part of that is they feel they worked harder, and another part of it is just that (i) they wanted it more (ii) they sacrificed more (iii) they had more fire in their eyes or whatever.

        Is such a person more likely to take a flyer on a Kovalev or a Grabovski? Can’t say for sure, but I’d be surprised.

      • STAN

        It drives me crazy when I hear the anyone from the Leafs talk about heart and compete level. Grabovski is a skilled player but those are the main reasons I always loved him.
        He worked his ass off in the offensive zone and was relentless pressuring the puck. He tried to fight the entire Habs team several times. He seemed to play better when the other team was targeting him physically, scoring the winning goal after being concussed by Chara being the prime example.
        Talking about heart and compete while simultaneously running a guy out of town who brings those things, as well as skill, to the weakest position in the organization never made any sense to me.

  • STAN

    The other “flaw” of analytics is that it doesn’t allow us to see what is going on the locker room and the coach himself. That is why we can’t really comment on Phaneuf’s leadership skills nor how Carlyle motivates and interacts with the players. We can only measure and judge the on ice product.

  • rw970

    We can keep right on Analyzing the Leafs all we want, but it hasn’t brought us a winner since 1967
    PS All Canadians should cheer the Habs the rest of the way.

  • STAN

    Forget on-ice analytics, how about some analysis of the Maple Leafs Empire Building. There is a clear and present reason that NO dominos have fallen in Toronto. No chips played.

    It’s called survival and enrichment inside a corporation with unlimited funds.

    Why on Earth would a brand new hockey team President spend even one Mulligan right off the hop when he’s been given carte blanche, an unlimited budget and probably a 10-year commitment by Leiweke and MLSE?

    He’ll save the Nonis/ Carlyle firings for 2016 or 2017, the scouting department overhaul for 2018-2020 and then set his sights on Leiweke’s gig when his 8-year contract expires in 2020. By then he can hire a new President that was also a star player like him. That guy might even have been a Leaf.

    We all seem to forget that Mr. Shanahan has zero executive experience. His NHL front office gig was looking at video and determining infractions and punishment. It was not rocket science and, BTW, we also seem to conveniently forget that he was pummelled for being inconsistency and for playing favourites. I would argue that the penalties meted by his successors have made far more sense.

    All we’ve seen regarding Shanahan’s business savvy, so far, is the instinct to survive. His willingness to give Nonis and Carlyle another chance helps insulate him and the new office he can gradually turn into a multi-employee, protective layer of management-for-the-sake-of-management.

    Once Nonis-Carlye is dispatched with fine parting gifts, Shanahan’s hand-picked replacements might be an improvement, but Leafs history (since 1967) says they won’t. He might then revamp the scouting department and their replacements may or m y not be more adept at finding talent.

    The fact that virtually all of Brian Burke’s Boys (Nonis, Carlyle, Captain Phaneuf, Kessel, Poulin, Loiselle, etc) remain and have gotten far richer than any of them deserves, speaks volumes – the OBC (Old Boys Club) is in full bloom in Toronto and is getting bigger, fatter and stupider.

    MLSE, as always, has far more money than brains.

  • STAN

    Props to MaxPawer417 and Jeremy Ian for visiting our HIO Habs site and telling us how many of you are pulling for us, and will be blogging with Habs fans throughout the series. Welcome all.

  • jasken

    Nice read Cam

    I really believe Shanny is eliminating the problems in the best possible manner according to evidence. The PP and PK and attack systems were addressed, he has made it so Carlyle takes responsibility along with the players.

    If he is right like I believe the attack left them weak defensively. It could fix a lot of issues, starting with high amount of shots against. He addressed the attacking which is a great start and buys him the time he needs to absorb and go through all the problems top to bottom.

    The fact that he is going slow addressing issues it might be 6 months or a year before he makes any big movements that might be needed. He is looking to future not the now he understands that sometimes the rash and hasty now decisions might be more costly in the future.

    He dont need a reason not to fire Carlyle, and he is not going to say all the inner turmoil and drama that might have been happening, even though we might enjoy it as it is none of our business.

    If whatever Shanny does starts to show positive results next season, he made the right moves. If it dont he fires Carlyle and continues on with more experience and addressing other issues.

  • jasken

    Good article. Time to focus on the positives here. The Leaf players have been put on notice for next year. The head coach has been put on notice. The farm team has prospects who are ready to challenge seriously for places on next year’s team.
    I’ll be extremely interested to see who the new assistant coaches are. I’ll also be interested to see the player moves that the Leafs make over the summer.
    This team can become very competitive very soon IMO.
    Training camp should be extremely competitive – can’t wait.