How the World Got Wired · Why Hinton Went North

The Long Quiet

The years that followed, for Geoffrey Hinton and for the field of neural networks more broadly, were lean.

The brief excitement of the 1986 backpropagation paper had not translated into widespread adoption. The computer hardware of the late 1980s was not powerful enough to train the kinds of large networks that the algorithm could in principle support. The networks people could actually train were small. They could learn small things. They were dramatically outperformed, in most practical applications, by other approaches — particularly by something called support vector machines, which were the fashionable thing in the 1990s.

Through the 1990s and into the early 2000s, what is now called the second AI winter descended on the field. Funding for AI research dropped sharply. Companies that had bet on artificial intelligence in the late 1980s went bankrupt. Most ambitious computer scientists moved into other areas — into the internet, into bioinformatics, into anything that was not AI.

Geoffrey Hinton kept working on neural networks.

He was joined, over the years, by a small number of others — a cluster of researchers, mostly in Canada, mostly funded by modest Canadian grants, who refused to give up on the approach. Yoshua Bengio, who had been Hinton’s PhD student and was now a professor in Montreal. Yann LeCun, a French researcher who had spent time with Hinton in Toronto in the late 1980s and then moved to Bell Labs in New Jersey. A handful of others.

Inside Canada, a charitable organization called the Canadian Institute for Advanced Research, or CIFAR, decided in 2004 to fund a long-term research program in what it called Neural Computation and Adaptive Perception. Hinton ran it. The budget was modest — about ten million Canadian dollars over five years, divided among a dozen researchers — but it was steady, and it had no military strings attached. CIFAR’s bet, simply put, was that the patient researchers who had stuck with neural networks through the winter would eventually be vindicated, and that Canada would benefit from being where they were when it happened.

Hinton, throughout this period, had a stream of brilliant graduate students. Most of them were not Canadian. They came from China, from India, from Russia, from Israel, from Europe, and from elsewhere in North America, drawn to Toronto by the simple fact that Hinton was there. Some of their names will mean something to you in a few years, if they don’t already. Ilya Sutskever. Alex Krizhevsky. Ruslan Salakhutdinov. These were Hinton’s PhD students in the 2000s. Each of them will reappear in the next chapter.

The work the group was doing through these years had no immediate practical application. It was the careful, mathematical, often frustrating work of trying to make large neural networks learn complicated things from data — something that, given the hardware available, kept almost working but not quite. They published papers that other researchers found interesting but that did not, for the most part, change anything in the world outside.

This is, I want to point out, what most science actually looks like. Most of it does not change the world. Most of it works on hard problems that may or may not be solvable, for decades. The people doing the work do not know, in advance, whether their work will turn out to matter. They do it because they think the problems are important, and because someone — a government, a university, a foundation — is willing to pay them, modestly, while they try.

For Geoffrey Hinton, the modest pay came from Canada. The patient time came from his own decision to spend his career on an unfashionable approach, in a country with smaller universities and less prestige than the American institutions he had left. The work, when it eventually paid off, would change everything. But during the long quiet years — and they were long, twenty years and more — almost nobody thought it would.

All 24 chapters of How the World Got Wired · Library · All courses