Abstract

Automatic lip-synchronous dubbing requires a speech synthesis model to generate alternating voice and silence patterns that match the timing of the original clip to ensure the best user experience. So far, this problem has been addressed by conditioning the speech synthesis process on lip movements extracted from the video signal. In this work, we condition speech generation on binary voice-activity signal, which has a lightweight representation and can be obtained in multiple ways, and show that the model follows the signal with high accuracy.

Examples from the Subjective Evaluation

Source Language	Source Audio	English Translation	Generated
French		only the denial of freedom. But it is also misery, it is also lack of hygiene, lack of care, overcrowding, and a whole load of things which are really inhuman.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Portuguese		I would like to end with this message from Madiba, that synthesizes all what I think as an educator.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Portuguese		Today I'm organizing a co-operative society where I want to give opportunity for other inmates, for young people who come from correctional institutes.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Greek		I wonder if we have ever recognized in ourselves reasons for someone to love us? And I don't mean a description like, "Tall, blonde, with blue eyes" or "Tall, strong, and manly" and these kinds of things	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Greek		What are the obligations of countries in which refugees arrive? They have the obligation to respect their right to live when it is perfectly clear that by sending them back home, they send them back to death.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Greek		I am already doing my fourth session without even being aware of it." I realized that she didn't want to talk, and I didn't want to be annoying.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Russian		If a reputable TV show hosts a biologist, it will definitely invite someone who stands against vaccination or GMOs, just to make sure we hear all sides.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)
Russian		And I discovered a newspaper called the "North Tallinn News", and there's a magazine "Health for All". They are very popular among babushkas, who tend to base their opinions on them.	LibriTTS (no VAD) LibriTTS (VAD) Multilingual (VAD)

Examples from the Qualitative Analysis

Scenario	Source Audio	Target Text	Generated
❌ One pause, comma too early		1 2 3 4 , 5 6 7 8 9 10
✅ One pause, comma in vicinity		1 2 3 4 5 , 6 7 8 9 10
✅ One pause, comma in vicinity		1 2 3 4 5 6 , 7 8 9 10
❌ One pause, comma too late		1 2 3 4 5 6 7 , 8 9 10
✅ One pause, few words		I like to eat fruit sometimes, like apples and pears.
✅ One pause, many words		I really do like to eat fruit from time to time, like for example apples and also pears.
❌ Two pauses, too many words, text only has one good pause location		I really like to eat fruit from time to time, like for example apples and also pears.
✅ Two pauses, even more words, but structure adapted to number of pauses		I really like to eat fruit sometimes, I really do enjoy it a lot, like for example apples and pears.