Abstract

When video is shot in noisy environment, the voice of a speaker seen in the video can be enhanced using the visible mouth movements, reducing background noise. While most existing methods use audio-only inputs, improved performance is obtained with our visual speech enhancement, based on an audio-visual neural network. We include in the training data videos to which we added the voice of the target speaker as background noise. Since the audio input is not sufficient to separate the voice of a speaker from his own voice, the trained model better exploits the visual input and generalizes well to different noise types. The proposed model outperforms prior audio visual methods on two public lipreading datasets. It is also the first to be demonstrated on a dataset not designed for lipreading, such as the weekly addresses of Barack Obama.

Interspeech 2018

Supplementary video

The following video contains a few examples of enhanced speech using our model.

Demos

#1
Dataset: Weekly addresses
Speaker: Obama
Noise: Trump's voice
[Noisy]
[Enhanced]

#2
Dataset: Weekly addresses
Speaker: Obama
Noise: Obama's own voice
[Noisy]
[Enhanced]

#3
Dataset: Weekly addresses
Speaker: Obama
Noise: Loud music
[Noisy]
[Enhanced]

#4
Dataset: GRID
Speaker: S15 (female)
Noise: S4 (female)
[Noisy]
[Enhanced]

#5
Dataset: GRID
Speaker: S2 (male)
Noise: S3 (male)
[Noisy]
[Enhanced]

#6
Dataset: TCD-TIMIT
Speaker: lipspkr2 (female)
Noise: Multiple speakers
[Noisy]
[Enhanced]

#7
Dataset: Mandarin
Speaker: Mandarin speaker
Noise: Multiple speakers and car engine
[Noisy]
[Hou et al.]
[Ours]

BibTeX

@inproceedings{gabbay2018visual,
  author    = {Aviv Gabbay and
	       Asaph Shamir and
	       Shmuel Peleg},
  title     = {Visual Speech Enhancement},
  booktitle = {Interspeech},
  pages     = {1170--1174},
  publisher = {{ISCA}},
  year      = {2018}
}
        

Code

Code for this work can be found here.

Related papers