Towards direct speech synthesis from ECoG: A pilot studyby C. Herff, G. Johnson, L. Diener, J. Shih, D. Krusienski, T. Schultz
Abstract:
Most current Brain-Computer Interfaces (BCIs) achieve high information transfer rates using spelling paradigms based on stimulus-evoked potentials. Despite the success of this interfaces, this mode of communication can be cumbersome and unnatural. Direct synthesis of speech from neural activity represents a more natural mode of communi- cation that would enable users to convey verbal messages in real-time. In this pilot study with one participant, we demonstrate that electrocoticography (ECoG) intracranial activity from temporal areas can be used to resynthesize speech in real-time. This is accomplished by reconstructing the audio magnitude spectrogram from neural activity and subsequently creating the audio waveform from these reconstructed spectrograms. We show that significant correlations between the original and reconstructed spectrograms and temporal waveforms can be achieved. While this pilot study uses audibly spoken speech for the models, it represents a first step towards speech synthesis from speech imagery.
Reference:
Towards direct speech synthesis from ECoG: A pilot study (C. Herff, G. Johnson, L. Diener, J. Shih, D. Krusienski, T. Schultz), In Engineering in Medicine and Biology Society (EMBC), 2016 38th Annual International Conference of the IEEE, 2016.
Bibtex Entry:
@INPROCEEDINGS{HerffEMBC2016,
author={Herff, C. and Johnson, G. and Diener, L. and Shih, J. and Krusienski, D. and Schultz, T.},
booktitle={Engineering in Medicine and Biology Society (EMBC), 2016 38th Annual International Conference of the IEEE},
title={Towards direct speech synthesis from ECoG: A pilot study},
year={2016},
url={https://www.csl.uni-bremen.de/cms/images/documents/publications/HerffEMBC_16.pdf},
poster={http://www.csl.uni-bremen.de/cms/images/documents/publications/Herff_EMBC16_poster.pdf},
month={Aug},
abstract={Most current Brain-Computer Interfaces (BCIs) achieve high information transfer rates using spelling paradigms based on stimulus-evoked potentials. Despite the success of this interfaces, this mode of communication can be cumbersome and unnatural. Direct synthesis of speech from neural activity represents a more natural mode of communi- cation that would enable users to convey verbal messages in real-time. In this pilot study with one participant, we demonstrate that electrocoticography (ECoG) intracranial activity from temporal areas can be used to resynthesize speech in real-time. This is accomplished by reconstructing the audio magnitude spectrogram from neural activity and subsequently creating the audio waveform from these reconstructed spectrograms. We show that significant correlations between the original and reconstructed spectrograms and temporal waveforms can be achieved. While this pilot study uses audibly spoken speech for the models, it represents a first step towards speech synthesis from speech imagery.}
}