The noose is tightening around Marty Schottenheimer's neck, even though the Chargers have retained his services for next season. According to some howling fans and nitpicking sportswriters, the Chargers coach chokes in playoff games. His regular season record -- 200 wins, 126 losses, 1 tie -- is one of the best in football history. But his playoff record is 5 wins and 13 losses. That's a major reason why the team owners, the Spanos family, got internal pressure to dump him.
But he has coached only 18 playoff games. Is that an adequate sample size -- enough games to merit this contumely? I interviewed eight statistical experts -- economists, financial gurus, physical scientists, mathematicians, statistical scientists. All work with numbers each day. Most have Ph.D.s. All but one are San Diegans. Only one works in sports. Since the playoffs involve tougher competition, and 18 games aren't enough to draw a conclusion, most said the numbers don't justify the opprobrium.
Mike Stolper of Stolper & Company makes a living sizing up the records of money managers. But there is a problem: "You need over 30 years for it to be statistically predictive," he says. "You never have statistical certainty because of the life span of human beings. Nobody gives you any responsibility until you are 40, and nobody trusts you after you are 60. People's life spans don't correspond to statistical purity." Thus, he says, "It is bizarre that they [fans, sportswriters, some in Chargers management] are focusing only on the [Schottenheimer] playoff record."
Stolper points to a famous mutual fund money manager who did better than the overall market for 15 years, until he took a pratfall in 2006 and significantly underperformed. Some statisticians said the odds of such a 15-year run were 1 in 2.3 million. But others pointed to statistical probability. With the thousands of portfolio managers out there, the odds that someone would make 15 in a row, perhaps mainly by luck, were probably 100 percent.
"We have modest statistical evidence that Marty's playoff performances are subpar," says Jeffrey Norman, executive vice president of Freeman Associates Investment Management. He got a bachelor's degree in math at Princeton and went on to postgraduate work in computer science and nightly plays high-stakes poker on the Internet. "We have nowhere near enough statistical evidence to infer whether this is due to Marty's coaching, his having to coach slightly weaker teams, or mere chance."
Adds Norman, "Playoff records really aren't that meaningful." If one coach has a 60 percent chance to win each playoff game and another coach has a 40 percent chance, "It would take 30 playoff games before you could tell with 90 percent confidence which coach is better."
With 18 games, "You don't have a big enough sample size to have a robust conclusion," says economist/lawyer/author Todd Buchholz, former White House and Harvard economist. The Schottenheimer condemnation makes Buchholz think of Yale economist Ray C. Fair, who believes that he can predict the winner of presidential elections by how fast the economy is growing. But Fair hasn't looked at enough years to achieve statistical reliability, says Buchholz. Ditto for the Schottenheimer posse.
Rodney Fort, professor of economics at Washington State University and author of Sports Economics (and the only non-San Diegan interviewed), believes the 18 games would have been significant if they had all come in succession and if each time Schottenheimer had had a team as good as the Chargers, who had a 14-2 regular season, the best in football, this season. In previous years, "He might have had marginal teams" compared to this season's Chargers, says Fort.
Jason Schweinsberg, a Ph.D. in statistics who is an assistant professor in the math department at the University of California, San Diego, says that people "attach too much significance to occurrences that could have easily happened by chance." For example, National Football League teams play only 16 regular season games. However, if all teams were equal and the games were decided by a coin flip, more than one team per year would have 12-4 records. "One team every three years would finish 13-3 or better just by luck," he says.
"Marty Schottenheimer's career playoff record is 5-13. However, if one flips a coin 18 times, one will get 5 or fewer heads nearly 5 percent of the time. While it is unlikely that a coach will go 5-13 just by having bad luck, it is certainly possible," says Schweinsberg, who spent three years doing postdoctorate work in math at Cornell. He notes that the Pittsburgh Steelers' coach, who just resigned, had lost all four American Football Conference championship games that he coached until last season. "That would happen by chance only about 6 percent of the time." Then the Steelers went on to win last year's Super Bowl.
Al Rappaport, emeritus professor of management from Northwestern University, now a prolific author for Harvard Business School Press and the Harvard Business Review, says he has spent his life examining statistics, but they don't explain everything. "I am a numbers guy," he says. "But to look at numbers only is probably unfair to everyone concerned. The coach calls the perfect play, but the guy drops the ball in the end zone, and the team loses the game." After the Chargers lost to New England January 14, players said the same thing.
"The playoffs filter the competition," says Rappaport. It's tougher to have a good record in playoffs.
As it turns out, the Patriots' coach, Bill Belichick, has an astonishingly good playoff record of 13 wins, 3 losses, although a less impressive 111-81 regular season record. Stuart Hurlbert, research professor of biology at San Diego State University and an expert in ecological statistics, disagrees with the others: even though there is a small sample size, the huge difference between the playoff records of Belichick and Schottenheimer is meaningful. "Statistically, that is a real difference," says Hurlbert. "I guess if people in San Diego had seen that coach's [Belichick's] amazing record, they would have been more worried. But once you get a little boosterism going, it is like a virus." Hurlbert also thinks there is a significant statistical difference between Schottenheimer's overall record and his playoff record.