He tweaked a machine learning technique called a hidden Markov model so it would work with fMRI data. The algorithm can be trained to recognize structure for events that stretch out over time, such as an episode of a TV show. The approach, borrowed from computer science, was new to neuroimaging. “That gave me a wedge,” Baldassano said. With this strategy he was able to sort through a massive data set: measures of neural activity from about 50,000 voxels across the brain, taken every 1.5 seconds during a 50-minute segment of Sherlock.
The model revealed a clear signal in the brain’s default mode network, which is thought to perform a range of cognitive functions involved in constructing internal narratives. A central hub in this network is the prefrontal cortex, which sits behind the forehead and attends to a person’s goals, plans and decisions. The network responds to meaningful shifts in a stimulus, such as a plot twist or a new topic of conversation. In the Sherlock data, dramatic fluctuations in activity occurred every minute or so, corresponding to what viewers perceived as scene changes.
“With Baldassano’s method, you could take that continuous brain data as people are watching a movie and look for where there are sudden changes in spatial activity patterns — and that matched what people would say were the boundaries in a movie,” Chen said. “It was a data-driven way of segmenting experience.”
The model revealed something else. The brain was not just segmenting at the boundaries people recognized as meaningful scene changes. Some parts of the brain subdivided the experience into shorter segments. In the visual system, shifts in activity occurred every second or so in response to changes in lighting and scenery. In mid-level visual regions that represent objects, brain activity shifted every 30 seconds or so to, say, track Cumberbatch as he moved across the screen.
The researchers proposed that the viewers combined these disparate narrative fragments into a coherent experience. Instead of memory being a province of the hippocampus along with a few other regions, as is commonly believed, Baldassano’s research suggested that memory formation involves collaboration among many brain regions.
“Memory is not one thing. It’s not a monolith,” Reagh said. “Your memory has a recipe for different pieces of the experience.”
However, Baldassano did not believe that these pieces are assembled anew with each life (or television) episode. “When an event starts, you already have some kind of scaffold,” he said. “It’s like a coloring book page that’s not colored in yet, but a lot of the lines are already there. When an event is happening, you are filling in the particular details.”
Many of our everyday experiences are repetitive: eating at a restaurant, visiting an airport, attending class, shopping for groceries. Baldassano believes that the brain builds a rich library of these “event scripts” that form scaffolds for memory.
Hatnim Lee for Quanta Magazine
Psychologists had formally proposed that such scaffolds existed in the 1970s. But the line of research had fallen out of favor, largely because no one had good tools for studying them in the brain. Baldassano now had those tools. And he had a clever idea for how to use movies and television to study the scripts on which we build our memories.
Following a Script
Just like life, movies tend to repeat common themes. We have romantic comedies and disaster flicks, superheroes and alien encounters, and across all of these genres there are commonalities, such as a meal in a restaurant or a race through an airport. Baldassano set out to take advantage of this redundancy to locate some pages of the brain’s coloring book.
While each of 33 study participants lay in an fMRI machine, he screened clips from eight movies: four set in restaurants (Brazil [1985], Derek [2008], Mr. Bean [1997] and Pulp Fiction [1994]) and four set in airports (Due Date [2010], Good Luck Chuck [2007], Knight and Day [2010] and Non-Stop [2014]). Every restaurant clip contained more or less the same sequence of events: Characters entered a restaurant, were seated, ordered and ate. Airport clips all showed people arriving at an airport, going through security, walking to and waiting at the gate, and boarding a flight. But the movies differed in their details: genres, actors, plot points.
The study generated reams of data, and Baldassano took a first crack at it himself, checking manually for patterns in brain activity at certain times in the airport scenes. He didn’t expect to see much — fMRI data is blurry and noisy, and there is plenty of activity that the technique can’t access. But he thought he had spotted something interesting. When he shared a slide of his results at a group meeting in 2017, Chen exclaimed: “Oh my gosh, it worked!”
“Once Janice told me that it worked, I was convinced that I hadn’t messed something up,” Baldassano recalled.
Then he let the computers run their analysis. When Baldassano plugged the data into his fMRI-optimized hidden Markov model, it revealed a definitive sequence of brain activation patterns that was shared across people and across movies for a given type of event. Across all restaurant clips, one pattern showed up when actors entered, which shifted to another when they were seated, yet again when they ordered food, and once more when the food arrived. All restaurant stories shared these four event patterns on average, with some story-unique differences added on top. The airport movies were similarly represented in the brain, with each step of the sequence characterized by a predictable cross-brain fingerprint centralized in the prefrontal cortex.