JMIR Publications

ABSTRACT

Background:

Electroencephalography (EEG) has been widely used to measure brain activity, but its potential to generate accurate images from neural signals remains a challenge. Most EEG decoding research has focused on tasks such as motor imagery, emotion recognition, and brainwave classification, which involve EEG signal analysis and classification. Although some studies have explored the correlation between EEG and images, focusing on EEG-image classification or transformation, leaving EEG-based image generation underexplored.

Objective:

The primary goal of this study was to extend EEG-based classification to image generation, addressing the limitations of previous methods and unlocking the full potential of EEG for image synthesis. To achieve more meaningful EEG-to-image generation, we developed a novel framework, NECOMIMI (NEural-COgnitive MultImodal EEG-InforMed Image Generation with Diffusion Models), which was specifically designed to generate images directly from EEG signals.

Methods:

We developed a two-stage NECOMIMI method, which integrated the NERV EEG encoder with a diffusion-based generative model. The Category-based Assessment Table (CAT) Score was introduced to evaluate the semantic quality of EEG-generated images. Additionally, the ThingsEEG dataset was employed to validate and benchmark the CAT Score, providing a standardized measure for assessing EEG-to-image generation performance.

Results:

The NERV EEG encoder achieved state-of-the-art performance (SOTA) in several zero-shot classification tasks, with an average accuracy of 94.8% in the 2-way task and 86.8% in the 4-way task, outperforming models like NICE, MUSE, and ATM-S. This highlighted its superiority as a feature extraction tool for EEG signals. In a one-stage image generation framework, EEG embeddings often resulted in abstract or generalized images, like landscapes, instead of specific objects. Our proposed two-stage NECOMIMI architecture effectively extracted semantic information from noisy EEG signals, showing its ability to capture and represent underlying concepts derived from brainwave activity.

Conclusions:

NECOMIMI demonstrated the potential of EEG-to-image generation, revealing the challenges of translating noisy EEG data into accurate visual representations. The introduction of the CAT Score provided a new evaluation metric, paving the way for future research to refine generative models. Additionally, the study highlighted the significant clinical potential of EEG-to-image generation, particularly in enhancing brain-machine interface systems and improving the quality of life for individuals with motor impairments. Clinical Trial: NA