|
Scientific Understanding of Consciousness |
Sleep Visual Imagery and Wakefulness Perception
Science 3 May 2013: Vol. 340 no. 6132 pp. 639-642 Neural Decoding of Visual Imagery During Sleep T. Horikawa, M. Tamaki, Y. Miyawaki, Y. Kamitani ATR Computational Neuroscience Laboratories, Kyoto 619-0288, Japan. 2Nara Institute of Science and Technology, Nara 630-0192, Japan. 3National Institute of Information and Communications Technology, Kyoto 619-0288, Japan. [paraphrase] Visual imagery during sleep has long been a topic of persistent speculation, but its private nature has hampered objective analysis. Here we present a neural decoding approach in which machine-learning models predict the contents of visual imagery during the sleep-onset period, given measured brain activity, by discovering links between human functional magnetic resonance imaging patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. Our findings demonstrate that specific visual experience during sleep is represented by brain activity patterns shared by stimulus perception, providing a means to uncover subjective contents of dreaming using objective neural measurement. Dreaming is a subjective experience during sleep often accompanied by vivid visual contents. Previous research has attempted to link physiological states with dreaming, but none has demonstrated how specific visual contents are represented in brain activity. The advent of machine-learning–based analysis allows for the decoding of stimulus- and task-induced brain activity patterns to reveal visual contents. We extended this approach to the decoding of spontaneous brain activity during sleep. Although dreaming has often been associated with the rapid-eye movement (REM) sleep stage, recent studies have demonstrated that dreaming is dissociable from REM sleep and can be experienced during non-REM periods. We focused on visual imagery (hallucination) experienced during the sleep-onset (hypnagogic) period (sleep stage 1 or 2) because it allowed us to collect many observations by repeating awakenings and recording participants’ verbal reports of visual experience. Reports at awakenings in sleep-onset and REM periods share general features such as frequency, length, and contents, while differing in several aspects, including the affective component. We analyzed verbal reports using a lexical database to create systematic labels for visual contents. We hypothesized that contents of visual imagery during sleep are represented at least partly by visual cortical activity patterns shared by stimulus representation. Thus, we trained decoders on brain activity induced by natural images from Web image databases. Three people participated in the functional magnetic resonance imaging (fMRI) sleep experiments, in which they were woken when an electroencephalogram signature was detected, and they were asked to give a verbal report freely describing their visual experience before awakening [duration, 34 ± 19 s (mean ± SD)]. We repeated this procedure to attain at least 200 awakenings with a visual report for each participant. On average, we awakened participants every 342.0 s, and visual contents were reported in over 75% of the awakenings. Offline sleep stage scoring further selected awakenings to exclude contamination from the wake stage in the period immediately before awakening (235, 198, and 186 awakenings for participants 1 to 3, respectively, used for decoding analyses) From the collected reports, words describing visual objects or scenes were manually extracted and mapped to WordNet, a lexical database in which semantically similar words are grouped as “synsets” in a hierarchical structure. Using a semantic hierarchy, we grouped extracted visual words into base synsets that appeared in at least 10 reports from each participant (26, 18, and 16 synsets for participants 1 to 3, respectively). The fMRI data obtained before each awakening were labeled with a visual content vector, each element of which indicated the presence or absence of a base synset in the subsequent report. We also collected images depicting each base synset from ImageNet, an image database in which Web images are grouped according to WordNet, or from Google Images, for decoder training. We constructed decoders by training linear support vector machines (SVMs) on fMRI data measured while each person viewed Web images for each base synset. Multivoxel patterns in the higher visual cortex [HVC; the ventral region covering the lateral occipital complex (LOC), fusiform face area (FFA), and parahippocampal place area (PPA); 1000 voxels], the lower visual cortex (LVC; V1 to V3 combined; 1000 voxels), or the subareas (400 voxels for each area) were used as the input for the decoders. To look into the commonality of brain activity between perception and sleep-onset imagery, we focused on the synset pairs that produced content-specific patterns in each of the stimulus and sleep experiments (pairs with high cross-validation classification accuracy within each of the stimulus and sleep data sets). The output scores for individual synsets showed diverse and dynamic profiles in each sleep sample. These profiles may reflect a dynamic variation of visual contents, including those experienced even before the period near awakening. Our findings provide evidence that specific contents of visual experience during sleep are represented by, and can be read out from, visual cortical activity patterns shared with stimulus representation. Our approach extends previous research on the (re)activation of the brain during sleep and the relationship between dreaming and brain activity by discovering links between complex brain activity patterns and unstructured verbal reports using database-assisted machine-learning decoders. The results suggest that the principle of perceptual equivalence, which postulates a common neural substrate for perception and imagery, generalizes to spontaneously generated visual experience during sleep. [end of paraphrase]
Return to — Sleep and Dreaming |