Towards Closed-Loop Speech Synthesis from Stereotactic EEG: A Unit Selection Approach
by , , , , , , , , , , ,
Abstract:
Neurological disorders can severely impact speech communication. Recently, neural speech prostheses have been proposed that reconstruct intelligible speech from neural signals recorded superficially on the cortex. Thus far, it has been unclear whether similar reconstruction is feasible from deeper brain structures, and whether audible speech can be directly synthesized from these reconstructions with low-latency, as required for a practical speech neuroprosthetic. The present study aims to address both challenges. First, we implement a low-latency unit selection based synthesizer that converts neural signals into audible speech. Second, we evaluate our approach on open-loop recordings from 5 patients implanted with stereotactic depth electrodes who conducted a read-aloud task of Dutch utterances. We achieve correlation coefficients significantly higher than chance level of up to 0.6 and an average computational cost of 6.6 ms for each 10 ms frames. While the current reconstructed utterances are not intelligible, our results indicate promising decoding and run-time capabilities that are suitable for investigations of speech processes in closed-loop experiments.
Download PDF
PDF URL: https://www.csl.uni-bremen.de/cms/images/documents/publications/Angrick_ICASP22.pdf
Reference:
Towards Closed-Loop Speech Synthesis from Stereotactic EEG: A Unit Selection Approach (Miguel Angrick, Maarten Ottenhoff, Lorenz Diener, Darius Ivucic, Gabriel Ivucic, Sophocles Goulis, Albert J. Colon, Louis Wagner, Dean J. Krusienski, Pieter L. Kubben, Tanja Schultz, Christian Herff), In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), volume , 2022.
Bibtex Entry:
@INPROCEEDINGS{angrick2022ClosedLoopUnitSelection,
  author={Angrick, Miguel and Ottenhoff, Maarten and Diener, Lorenz and Ivucic, Darius and Ivucic, Gabriel and Goulis, Sophocles and Colon, Albert J. and Wagner, Louis and Krusienski, Dean J. and Kubben, Pieter L. and Schultz, Tanja and Herff, Christian},
  booktitle={ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  title={Towards Closed-Loop Speech Synthesis from Stereotactic EEG: A Unit Selection Approach},
  year={2022},
  volume={},
  number={},
  pages={1296-1300},
  abstract={Neurological disorders can severely impact speech communication. Recently, neural speech prostheses have been proposed that reconstruct intelligible speech from neural signals recorded superficially on the cortex. Thus far, it has been unclear whether similar reconstruction is feasible from deeper brain structures, and whether audible speech can be directly synthesized from these reconstructions with low-latency, as required for a practical speech neuroprosthetic. The present study aims to address both challenges. First, we implement a low-latency unit selection based synthesizer that converts neural signals into audible speech. Second, we evaluate our approach on open-loop recordings from 5 patients implanted with stereotactic depth electrodes who conducted a read-aloud task of Dutch utterances. We achieve correlation coefficients significantly higher than chance level of up to 0.6 and an average computational cost of 6.6 ms for each 10 ms frames. While the current reconstructed utterances are not intelligible, our results indicate promising decoding and run-time capabilities that are suitable for investigations of speech processes in closed-loop experiments.},
  keywords={},
  doi={10.1109/ICASSP43922.2022.9747300},
  url={https://www.csl.uni-bremen.de/cms/images/documents/publications/Angrick_ICASP22.pdf},
  ISSN={2379-190X},
  month={May},}