Silent Speech Communication

SilentSpeechArrayLorenzThe Cognitive Systems Laboratory at the University of Bremen is working on innovative Silent Speech Interfaces, which allow humans to communicate with each other by speaking silently. In march 2010, we demonstrated a prototype system at CeBIT, the world's largest IT fair, for the first time. We also presented our technology at the CeBIT Vision press show on November 24, 2009.

Technology

Our technology is based on Electromyography, i.e. the capturing and recording of electrical potentials that arise from muscle activity. Speech is produced by the contraction of muscles that move our articulatory apparatus. The electric potentials which are generated by this muscular activity are captured by surface electrodes attached to the skin. The analysis and processing of these signals by suitable pattern matching algorithms allow the reconstruction of the corresponding movement of the articulatory muscles and to deduct what has been said. The recognized speech is output as text or synthesized as an acoustic signal. Since electromyography records the muscle activity rather than acoustic signals, speech can be recognized even if it is uttered silently, without any sound production.

Applications

Our group has been working on this technology since 2004; a prototype was presented at CeBIT 2010 for the first time, showing the following applications:

  1. Silent Telephony: Silent speech recognition allows for silent communication without disturbing any bystanders.
  2. Transmitting Confidential Information: The system allows for seamless switches between silent and audibly spoken speech and thus enables to safely and securely transmit confidential information such as passwords and PINs.
  3. Robust communication in adverse environments: Since electromyography relies on signals captured directly at the human body, the signal is not corrupted by noisy and adverse conditions.
  4. Speaking in a foreign tongue: By feeding the output of silent speech recognition into a component that translates from one language to another, native speakers can silently utter a sentence in their language, and the receivers hear the translated sentence in their language. It appears as if the native speaker produced speech in a foreign language.
  5. Help for disabled people: Our technology may also help people who have lost their voice due to accident or illness.

Downloads

EMG-UKA Trial Corpus

Contact

Please direct any inquiries to the following people:

Prof. Dr.-Ing. Tanja Schultz
Cognitive Systems Lab, Universität Bremen
Enrique-Schmidt-Str. 5
28359 Bremen
Tel. +49 (0) 421 218 64270

Lorenz Diener M. Sc.
Cognitive Systems Lab, Universität Bremen
Enrique-Schmidt-Str. 5
28359 Bremen
Tel. +49 721/608-45059

Relevant Publications

Below, you can find a list of our publications related to Silent Speech Communication, sorted by publication date. Two papers that provide a good introduction to and overview of the topic are Session-Independent EMG-based Speech Recognition (, , International Conference on Bio-inspired Systems and Signal Processing, ) and Modeling Coarticulation in EMG-based Continuous Speech Recognition (, , Speech Communication Journal, volume 52, ). 

2016
[45] Towards direct speech synthesis from ECoG: A pilot study (, , , , , ), In Engineering in Medicine and Biology Society (EMBC), 2016 38th Annual International Conference of the IEEE, . [bibtex] [abstract] [pdf] [poster]
[44] An Initial Investigation into the Real-Time Conversion of Facial Surface EMG Signals to Audible Speech (, , , ), In Engineering in Medicine and Biology Society (EMBC), 2016 38th Annual International Conference of the IEEE, . [bibtex] [abstract] [pdf] [poster]
2015
[43] Direct Conversion from Facial Myoelectric Signals to Speech using Deep Neural Networks (, , ), In International Joint Conference on Neural Networks, . (IJCNN 2015) [bibtex] [abstract] [pdf] [doi]
[42] Codebook Clustering for Unit Selection Based EMG-to-Speech Conversion (, , ), In Sixteenth Annual Conference of the International Speech Communication Association, . (Interspeech 2015) [bibtex] [abstract] [pdf]
2014
[41] Conversion from Facial Myoelectric Signals to Speech: A Unit Selection Approach (, , , ), In The 15th Annual Conference of the International Speech Communication Association, Singapore, . (Interspeech 2014) [bibtex] [abstract] [pdf]
[40] Towards Real-life Application of EMG-based Speech Recognition by using Unsupervised Adaptation (, ), In The 15th Annual Conference of the International Speech Communication Association, Singapore, . (Interspeech 2014) [bibtex] [pdf]
[39] The EMG-UKA Corpus for Electromyographic Speech Processing (, , ), In The 15th Annual Conference of the International Speech Communication Association, Singapore, . (Interspeech 2014) [bibtex] [abstract] [pdf]
[38] Tackling Speaking Mode Varieties in EMG-Based Speech Recognition (, ), In IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, volume 61, . [bibtex] [abstract]
[37] Pattern Learning with Deep Neural Networks in EMG-based Speech Recognition (, ), In 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, . (EMBC 2014) [bibtex] [pdf]
[36] Compensation of Recording Position Shifts for a Myoelectric Silent Speech Recognizer (, , , ), In The 39th International Conference on Acoustics, Speech, and Signal Processing, . (ICASSP 2014) [bibtex] [abstract] [pdf]
[35] BioKIT - Real-time Decoder For Biosignal Processing (, , , , , , , , , , , ), In The 15th Annual Conference of the International Speech Communication Association, Singapore, . (Interspeech 2014) [bibtex] [abstract] [pdf]
[34] Enhancement of EMG-based Thai Number Words Classification using Frame-based Time Domain Feature with Stacking Filter (, , , , , ), In Proceedings of 2014 APSIPA Annual Summit and Conference, . [bibtex] [abstract]
[33] Fundamental Frequency Generation for Whisper-to-Audible Speech Conversion (, , , , ), In The 39th International Conference on Acoustics, Speech, and Signal Processing, . (ICASSP 2014) [bibtex] [abstract] [pdf]
[32] Spatial Artifact Detection for Multi-Channel EMG-Based Speech Recognition (, , , ), In 7th International Conference on Bio-inspired Systems and Signal Processing, . (BIOSIGNALS 2014) [bibtex] [abstract] [pdf]
2013
[31] Artifact Removal Algorithm for an EMG-based Silent Speech Interface (, , , , ), In International Conference of the IEEE Engineering in Medicine and Biology Society, Osaka, Japan, . (EMBC 2013) [bibtex] [abstract] [pdf]
[30] Array-based Electromyographic Silent Speech Interface (, , , ), In 6th International Conference on Bio-inspired Systems and Signal Processing, . (BIOSIGNALS 2013) [bibtex] [abstract] [pdf]
[29] Application of Electrode Arrays for Artifact Removal in an Electromyographic Silent Speech Interface (, , , , , , ), In Biomedical Engineering Systems and Technologies International Joint Conference, Barcelona, Spain (Also in: Revised Selected Papers Communications in Computer and Information Science, Vol. 452), . (BIOSTEC 2013) [bibtex] [abstract] [pdf]
[28] Biosignale-basierte Mensch-Maschine-Schnittstellen (, , , , ), In at - Automatisierungstechnik, 2013, volume 61, . [bibtex] [pdf]
2012
[27] Decision-tree based Analysis of Speaking Mode Discrepancies in EMG-based Speech Recognition (, , ), In International Conference on Bio-inspired Systems and Signal Processing, . (BIOSIGNALS 2012) [bibtex] [abstract] [pdf]
[26] Inferring Prosody from Facial Cues for EMG-based Synthesis of Silent Speech (, , , ), In 4th International Conference on Applied Human Factors and Ergonomics, . [bibtex] [abstract] [pdf]
[25] Further Investigations on EMG-to-Speech Conversion (, , , ), In IEEE International Conference on Acoustics, Speech and Signal Processing, . (ICASSP) [bibtex] [abstract] [pdf]
2011
[24] Session-Independent EMG-based Speech Recognition (, ), In International Conference on Bio-inspired Systems and Signal Processing, . [bibtex] [pdf]
[23] Analysis of Phone Confusion in EMG-based Speech Recognition (, ), In IEEE International Conference on Acoustics, Speech and Signal Processing, . [bibtex] [pdf]
[22] Estimation of Fundamental Frequency from Surface Electromyographic Data (, , , ), In IEEE International Conference on Acoustics, Speech and Signal Processing, . [bibtex] [abstract] [pdf]
[21] Investigations on Speaking Mode Discrepancies in EMG-based Speech Recognition (, , ), In 12th Annual Conference of the International Speech Communication Association, . (Interspeech 2011) [bibtex] [abstract] [pdf]
[20] Impact of Different Feedback Mechanisms in EMG-based Speech Recognition (, , , ), In 12th Annual Conference of the International Speech Communication Association, . (Interspeech 2011) [bibtex] [abstract] [pdf]
2010
[19] Speaker-Adaptive Speech Recognition Based on Surface Electromyography (, ), In Biomedical Engineering Systems and TechnologiesInternational Joint Conference, BIOSTEC2009, Porto, Portugal, January 14-17, 2009, Revised Selected PapersCommunications in Computer and Information Science , Vol. 52, . [bibtex] [pdf]
[18] Modeling Coarticulation in EMG-based Continuous Speech Recognition (, ), In Speech Communication Journal, volume 52, . [bibtex] [pdf]
[17] Spectral Energy Mapping for EMG-based Recognition of Silent Speech (, , ), In First International Workshop on Bio-inspired Human-Machine Interfaces and Healthcare Applications, . (Side event of Biosignals 2010 conference) [bibtex] [abstract] [pdf]
[16] Impact of Lack of Acoustic Feedback in EMG-based Silent Speech Recognition (, , ), In 11th Annual Conference of the International Speech Communication Association, Makuhari, Japan, . (Interspeech 2010) [bibtex] [abstract] [pdf]
[15] BiosignalsStudio: A flexible Framework for Biosignal Capturing and Processing (, , , , , , ), In 33rd Annual German Conference on Artificial Intelligence 2010, . [bibtex] [pdf]
2009
[14] Towards Speaker-Adaptive Speech Recognition based on Surface Electromyography (, ), In 2nd International Conference on Bio-inspired Systems and Signal Processing, Porto, Portugal, . (Biosignals 2009) [bibtex] [pdf]
[13] Impact of Different Speaking Modes on EMG-based Speech Recognition (, , , ), In 10th Annual Conference of the International Speech Communication Association, . [bibtex] [pdf]
[12] Synthesizing Speech from Electromyography using Voice Transformation Techniques (, , ), In 10th Annual Conference of the International Speech Communication Association, . [bibtex] [pdf]
2008
[11] Automatic Speech Recognition based on Electromyographic Biosignals (, ), In Selected from the BIOSTEC full papers to be published in:Communications in Computer and Information Science (CCIS) series publishedby Springer, . [bibtex] [pdf]
[10] EARS: Electromyographical Automatic Recognition of Speech (, ), In 1st International Conference on Bio-inspired Systems and Signal Processing, Madeira, Portugal, . (Biosignals 2008) [bibtex] [pdf]
2007
[9] Wavelet-based Front-End for Electromyographic Speech Recognition (, , ), In Proceedings of Interspeech, . [bibtex] [pdf]
[8] Continuous Electromyographic Speech Recognition with a Multi-Stream Decoding Architecture (, , ), In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, . [bibtex] [pdf]
[7] Whispering Speaker Identification (, , ), In Proceedings of ICME, . (Multimedia Interaction Human Machine Interface) [bibtex] [pdf]
2006
[6] Sub-Word Unit based Non-audible Speech Recognition using Surface Electromyography (, , , , ), In In proceedings of the 9th ISCA International Conference on Spoken Language Processing, . [bibtex] [pdf]
[5] Towards Continuous Speech Recognition Using Surface Electromyography (, , , , ), In In proceedings of the 9th ISCA International Conference on Spoken Language Processing, . [bibtex] [pdf]
[4] Articulatory Feature Classification using Surface Electromyography (, , ), In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, . [bibtex] [pdf]
2005
[3] Whispery Speech Recognition using Adapted Articulatory Features (, , ), In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, . [bibtex] [pdf]
2004
[2] Using Word Lattice Information for a Tighter Coupling in Speech Translation Systems (, , , ), In International Conference of Spoken Language Processing, . [bibtex] [pdf]
[1] Adaptation for Soft Whisper Recognition Using a Throat Microphone (, , ), In International Conference of Spoken Language Processing, . [bibtex] [pdf]

 

CSL Logo rot webseite 

 
Contact
Cognitive Systems Lab
Prof. Dr.-Ing. Tanja Schultz
 
Enrique-Schmidt-Str. 5
28359 Bremen
Germany

Phone: +49 (0) 421 218 64270
E-Mail: