Descript Revolutionizes Multilingual Video Dubbing with OpenAI's AI

Descript: AI at the Heart of Multilingual Video Editing

Descript, an innovative video editor, has integrated artificial intelligence to transform the way videos are edited and translated. The founding idea is simple: if you can edit text, you should be able to edit videos with the same ease. Since its inception, Descript has utilized AI for every aspect of its product, from transcription to editing, including audio cleanup. The company has partnered with OpenAI, leveraging tools like Whisper for transcription and GPT models in their co-editor Underlord to enhance its features.

Video Translation: A Challenge Simplified by AI

Video translation has long been a complex and costly process, requiring linguistic experts to ensure accurate and high-quality translations. However, the emergence of large language models (LLMs) has simplified this process. These models compress the translation steps, making high-quality translation possible on an unprecedented scale.

Subtitles and Dubbing: The Quest for Semantic Fidelity

When it comes to subtitling and dubbing, semantic fidelity is crucial. The translation must not only preserve the original meaning but also adhere to timing constraints. For subtitles, this temporal adherence is an asset, but for dubbing, it is essential. A translated speech that does not respect the original duration can seem artificial, even if the meaning is correct.

Notable Improvements Thanks to AI

To overcome these challenges, Descript has rethought its translation process using OpenAI's reasoning models. This has allowed for the simultaneous optimization of semantic fidelity and timing adherence during generation, rather than as an afterthought. In just 30 days after deploying these improvements, exports of translated videos with dubbing increased by 15%, and adherence to timing improved by 13 to 43 percentage points depending on the languages.

Dubbing: A Growing Demand

Dubbing has become an increasingly popular feature for Descript. According to Laura Burkhauser, CEO of Descript, the company is developing solutions to enable businesses to translate and lip-sync entire libraries of video content.

Initial Challenges of Dubbing

Initially, translation was one of the most requested features by Descript users. The company started by translating only subtitles, but many users also wanted audio dubbing in the target language. However, a major issue arose: the dubbed audio did not always sound natural. Aleks Mistratov, Head of AI Product at Descript, explained that the speech rhythm was often unnatural in the translated language.

Linguistic Differences: An Obstacle to Natural Dubbing

The difficulty lies in the fact that different languages require different durations to express the same idea. For example, German is generally longer than English. To fit fixed video segments, the translated speech often had to be artificially sped up or slowed down, which could result in an unnatural outcome.

Solutions Proposed by Descript

To address this issue, users had two options: manually adjust the timing of the audio or rewrite the translation for better fit. These solutions required deep adjustments and near-native proficiency in the target language, making the process tedious for creators and limiting the extension of this feature to large localization projects.

Towards an Optimization of Timing and Meaning

The Descript team understood that to succeed in dubbing, it was necessary to optimize not only semantic meaning but also to respect timing constraints. When translating from English to German, for example, the model needed to be able to simplify the concept or use fewer words so that the dubbed audio remained natural.