The Question
How do we see in 3D? How our brain does this remains a mystery. All the brain receives are two 2D images on the retina, and yet it somehow constructs a single unified 3D percept. While we know how we would design a visual system with these inputs, the reality is that human 3D vision doesn’t behave as we would expect.
Solving this question has a number of significant implications. First, it helps us to understand the human brain. 3D vision is one of the classic paradigms through which general principles about the brain are formulated and tested, from single-cell recordings in the 1960s, to computational approaches in the 1980s, to probabilistic models in the 1990s. Second, public health. Roughly a billion people worldwide (one-eighth of the world’s population) have 3D vision deficits, so understanding the mechanisms underpinning 3D vision is key. Third, artificial intelligence. We are able to act and navigate in the 3D world in a way that far surpasses existing AI with a fraction of the computational power. How does the human brain do it? Fourth, virtual and augmented reality. We can only build a convincing virtual replica of reality if we know what is required to convince human vision in the first place. Fifth, art, architecture, and spatial design. They all appeal to principles of how we perceive the 3D world.
The Approach
The New Approach to 3D Vision project seeks to unravel the seemingly simple question of how we see in 3D. To understand any system, it is helpful to understand what inputs it responds to. The 3D Vision researchers will thus begin by testing how different visual inputs lead to different 3D visual experiences through behavioral experiments. For example, 3D vision isn’t informed by eye rotation, a mechanism that was thought to be important for linking the retinal image to the world. More recently, the project’s principal investigator developed the “Linton Stereo Illusion” to show that 3D vision directly responds to the projections on the retina rather than reconstructing the position of points in the world.
Once these distortions of visual space have been appropriately quantified, the researchers will build computational models of human 3D vision, collaborating with Nikolaus Kriegeskorte at Columbia’s Zuckerman Mind Brain Behavior Institute.
Finally, they will look for evidence of these processes at work in specific regions of the brain. Bringing together 14 researchers from the US and Europe, the 3D Vision PI will lead a Generative Adversarial Collaboration titled “Is V1 a Cognitive Map?”
The New Approach to 3D Vision project is being led by Paul Linton at the Italian Academy for Advanced Studies, Columbia University (New York, US).
Feature image: Ann Veronica Janssens: yellowbluepink exhibition, Thomas SG Farnetti. (Source: Wellcome Collection [CC BY-NC 4.0])
