Chapter 8: How to Explain a Brain

How DiCarlo decided to pivot from conventional modeling methods to the modeling methods of artificial neural networks, AI methods, considered to be unrealistic

Note: This chapter is part of the proof-of-concept material for a book, AI: How We Got Here—A Neuroscience Perspective. If you like it, please consider supporting the Kickstarter project.

DiCarlo describes the state of vision research in this period as somewhat stagnant. Perhaps it would even be fair to say it was congealed and backed up. He was a young, ambitious neuroscientist, and he wanted to make progress; but it wasn’t clear how to do that.

When it came down to it, vision neuroscientists knew a lot about the mind, and a lot about the visual cortex. They had created a vast literature of studies; studies of various microscopic phenomena; of different anatomical characteristics; of different brain regions and subregions; taken while organisms performed different visual tasks; taken using different methods; and so on. But they were like Charles Darwin, sailing on the Beagle, before he came up with his theory of evolution; seeing lots of things about the world that seemed to be related, without it making sense yet. For all their knowledge, they couldn’t explain the way the brain worked.

But really, what did it mean to explain the brain, in this case? Mostly, it meant to explain why the neurons in the brain fired the way they did, and how they collectively gave rise to behaviors. So what was the problem? Well, they could explain why and how single neurons fired, using models like Hodgkin’s and Huxley’s.

But, there was a caveat. Recall that these models, like all models, served in two different fashions. First, they consisted in an equation (or equations) that served as a model of a phenomenon, in general; the equation represented a neural signal with one variable, and related it to other variables (with equal signs, plus signs, multiplication, and so on), which represented other, measurable characteristics, like a neuron’s cell membrane capacitance, or the membrane’s proclivity for sodium and calcium flows. Second, the equation could serve as a model of a specific neural signal; if you had measured the value of those other variables for a specific neuron, you could solve the equation for the signal you expected to measure in it. Conversely, if you hadn’t measured them, you couldn’t solve for the signal, and you therefore couldn’t explain the value you had measured for it, on a quantitative level.

For DiCarlo to explain the actual neural signals he had measured in his laboratory experiments, he needed to use models like Hodgkin and Huxley’s in their second fashion, as models of specific neural signals. The problem was that neuroscientists didn’t know how to collect all the data needed to solve these models. The scale of the visual cortex was staggering, even though it was only a fraction of the nervous system. The visual cortex of the monkey, for example, by itself, contains roughly two hundred fifty million neurons per brain hemisphere. That meant that there was a metric asston of specific variables you would have to measure, to build a bottom-up model of the entire visual cortex brain region.

To some extent, this was the same problem faced by the OpenWorm collaborators. They had models for the neurons, but they did not have measurements of all the characteristics needed to solve the models. However, because the worm possessed only 302 neurons, the amount of data needed was not unmanageable. The greater impasse was the need for a great many different models or equations; studies had shown that many neurons in the worm functioned in very different ways, for example, geared specifically for detecting smells; for regulating certain hormones; for accomplishing locomotion, and so on. In short, whereas the OpenWorm project was starved of models (or equations), vision neuroscientists were starved of data.

Vision neuroscientists like DiCarlo were therefore left with two alternatives. First, they could cling to a focus on understanding neural signals, without insisting on explaining those signals with bottom-up models. For example, they might show that some of the neurons responded more to curves versus non-curves, or colors versus black-and white features. DiCarlo did something like this with his 2005 study, which showed conclusively that IT neurons responded to objects. These were important findings, but but it was hard to imagine how you would ever explain the visual cortex, in a holistic way, based on hundreds of such separate fact finding missions.

The second alternative for neuroscientists was to focus on understanding behaviors, like object recognition, without linking them to neural signals. Much headway could be made with this approach, since behaviors were so complicated, and there was so much to learn about behavioral details. But it was also an approach that largely abandoned the chief goals of neuroscience, which was to link behaviors to their underlying neural signals.

Trained as both a neuroscientist and engineer, this dilemma frustrated DiCarlo. Because the first approach of focusing on neural signals was more popular, it had so far shaped most of his own research. The second approach, focusing on behaviors, without linking them to neural signals, seemed to painfully irrelevant to neuroscience that by the early-2000s, it had almost mitosised itself off of the field, entirely. It had come to be embraced much more by other specialists, like cognitive scientists, psychologists, and even computer scientists, who all mainly cared about behaviors, and didn’t trouble themselves with the seeming electrical haywire of neural signals.

But in a strange twist, the primary methods of understanding behaviors, adopted by these other specialties, was based on a certain classic kind of bottom-up models of neurons. They were first invented in the 1940s, by Warren McCulloch and Walter Pitts, well before Hodgkin and Huxley came up with a first realistic model of a single neuron. Rather than being intended to explain neural signals, they were more intended to explain how neural ensembles could give rise to behaviors. Their equations were highly abstract, distilling the complexity of real biological neurons into just a couple variables, and they put little emphasis on actually knowing those variables, or using measured data. Their models were therefore considered unrealistic, even though paradoxically, they were created with a standard approach to creating realistic models.

Nonetheless, although neural networks were mainly being studied outside of neuroscience, much progress had been made with using them. In fact, so much progress had been made that it was starting to come full circle, starting to make them more relevant for neuroscience. So, when DiCarlo gained tenure in 2008, he decided to break away from the bulk of his field, which had abandoned the aspiration for realistic models. “I kind of knew this was not the way it was going to work,” he told me. “We were going to have to build some serious models.”