Neural decoding of visual information across different neural recording modalities and approaches

Every day, various types of sensory information from the external environment are transferred to the brain through different modalities and then processed to generate a series of coping behaviors. Among these perceptual modalities, vision is arguably the dominant contributor to the interactions between the external environment and the brain.

Approximately 70% of human perception information is derived from vision, far more than the auditory system, tactile system, and other sensory systems combined. The visual system is the part of the central nervous system that is required for visual perception, processing, and interpreting visual information to build a representation of the visual environment.

It consists of the eye, retina, fibers that conduct visual information to the thalamus, the superior colliculus, and parts of the cerebral cortex. Today, researchers can use different recording modalities—for example, spikes, electroencephalography (EEG), and functional magnetic resonance imaging (fMRI)—to collect neural signals from brain activity in different parts of the visual system, such as the retina, lateral geniculate nucleus (LGN), and primary visual cortex (V1) cortex, etc.

Depending on the corresponding collecting devices, different recording modalities differ in their invasiveness, scale, and precision.

Neural coding is an important topic for understanding how the brain processes stimuli from the environment. The aim of neural decoding is to read out information embedded in various types of neural signals. As for vision, understanding how neurons perceive and respond to rich natural visual information is a major topic of neural encoding, whereas the goal of neural decoding of visual information is to restore the original stimulus from neural responses as much as possible.

It is also critical to the development of artificial vision used by brain-computer interfaces and virtual reality devices.

Much effort has been made to study the various mechanisms underlying neural decoding in the visual pathway in recent decades. These mechanisms can be roughly divided into three categories depending on the decoding type:

1. Visual stimulus classification, in which a specific stimulus is classified into the best-matched image set
2. Visual stimulus identification, in which the stimulus is identified with a specific visual object
3. Visual stimuli reconstruction, in which the corresponding visual stimulus is reconstructed in accordance with the resulting neural responses.

Most decoding approaches have depended nonlinear methods due to their interpretability and computational efficiency. Although linear decoding methods are capable of decoding spatially uniform white noise stimuli and the coarse structure of natural scene stimuli from neural responses, the recovery of the fine visual details of naturalistic images is difficult for these types of methods. The most recent decoders have utilized nonlinear methods for the fine decoding of complex visual stimuli.

For instance, optimal Bayesian decoding was leveraged for white noise stimuli but achieved limited generalizability to a large neural population. For natural scene image structures, key prior information was used to perform computationally expensive approximations to Bayesian inference. Some researchers have combined linear and nonlinear approaches to generate coarse reconstructions of natural stimuli through calcium imaging data. Additionally, many researchers have begun to successfully use deep learning techniques for visual neural decoding, leading to great achievement in artificial vision.

Visual neural decoding is a significant issue that can help advance engineering applications such as brain–machine interfaces and a more holistic understanding of the brain in neuroscience. Considering the rapid developments of related techniques in visual neural decoding, there is a strong demand for a comprehensive and up-to-date review in this field.

In a new review published in the journal Machine Intelligence Research, researchers have sorted out the research evolution in visual neural decoding. Various neural recording modalities are introduced in this review, especially for the emerging calcium imaging data.

The researchers' aim was to provide a review of neural decoding in visual systems that could serve as an inspiration to both neuroscience and multidisciplinary researchers looking to understand the state-of-the-art and current problems in neural decoding, especially regarding the development of artificial intelligence and brain-like vision systems.

In this paper, the team summarized the advantages and disadvantages of different neural decoding methods. In addition, open resources, including public, neural data and software toolkits, are also provided for the convenience of neural decoding research. Finally, they concluded with their perspective on the open challenges and future directions for the outlook in this study.

The researchers first briefly analyzed the evolution of decoding tasks, i.e., classification, identification, and reconstruction, as this research field has developed. Then they introduced the main neural recording modalities used in visual neural decoding, including spikes, EEG, fMRI, and calcium imaging signal, and analyzed the characteristics of the data they acquired. Figures in this paper show the characteristics of the signals obtained with neural recording modalities and summarize the differences in signal types, data structures, and spatial and temporal resolutions.

Then, in Section 4, the team reviewed the main types of decoding approaches that have been proposed in recent decades in this field. The first is linear decoding methods. Most early approaches to visual neural decoding have depended on linear methods due to their interpretability and computational efficiency. Nevertheless, the limited representation power of linear methods makes it difficult to reconstruct the fine visual details of natural images.

The second is Bayesian-based decoding methods. Bayesian decoding models usually outperformed simple linear decoding models in visual neural decoding tasks. However, there are also some constraints for Bayesian decoding methods. The Bayesian decoding methods usually have to resort to the specific prior information encoded by a specifically designed model. The determination of parameters in the overall decoding process needs to be elaborated.

Furthermore, the mapping between the visual stimuli and the corresponding brain activity determined by Bayesian methods does not typically describe the relationship between these two cross-modal data. Consequently, fine natural image details are difficult to reconstruct with this type of method. The third is deep neural network methods, which include the introduction of CNN, RNN, generative adversarial networks (GANs), transfer learning, and so on.

The open-source nature of datasets with high-quality neuronal physiological responses is essential for neuroscience research. These open-source data objectively connect different research works through sharing data and play an important role in forming benchmarks in the field of neural decoding. In Section 5, researchers summarized three large-scale open neural databases widely used worldwide, as well as some neural analysis software toolboxes.

The issue of visual neural decoding has been a topic of interest for decades, with rapid advances in the development of both brain-activity recording techniques and neural decoding analysis methods. In Section 6, the researchers highlighted several potential directions and open challenges, and hope to provide other researchers with insight into this issue.

The ultimate purpose of visual decoding is to decode the content of people's experiences in the absence of visual input. However, the scarcity of pairwise neurophysiological stimulus datasets and accurate, large-scale recording neural modalities continue to hinder the development of this discipline.

Nevertheless, the importance of visual neural decoding cannot be understated. The development of neural decoding technology will promote the development of neural prostheses and brain-computer interfaced devices. The researchers hope that their brief review will inspire ideas for future work in the cross-disciplinary field of brain science and neural computing.