Microsoft AI May Soon Change What People Say In Live Videos: Here's How

Imagine watching a live speech in which the speaker's words are getting changed as they speak. Yes, you read that right. This may be possible in the future as Microsoft is exploring a technology that could use AI to replace specific words and phrases in a live video while also changing the speaker's mouth movements and facial appearance to match.

The idea comes from a newly filed Microsoft patent. It describes a system that could edit audio and video in near real time, making the changes appear as though the speaker had said the altered words in the first place. The system could also follow predefined instructions or individual user profiles, meaning different viewers could potentially see different versions of the same broadcast.

Microsoft has not announced plans to turn the patent into a product. However, the proposed system goes beyond subtitles and audio filters, with the ability to modify the video itself while it is playing.

Microsoft AI Patent Could Change Live Translation And Speech

The patent describes AI agents that can identify specific words or phrases and decide whether they should be removed or replaced. Once the audio is changed, the system would adjust the speaker's lip movements and facial appearance to match. It could also edit the surrounding audio and video to prevent sudden cuts or noticeable interruptions.

One possible use is live language translation. A viewer could watch a speech in their preferred language, with the speaker's mouth movements changed to match the translated audio. The technology could also remove filler words such as “um” and “ah”, unnecessary pauses and stutters, making a speech easier to follow.

It could even correct mispronounced or incorrectly used words without asking the speaker to repeat them. But the patent describes more than just making speech clearer.

The system could replace colloquial phrases, coined terms and local expressions with different words. For example, someone using “y'all” or “bite the bullet” during a presentation could have those phrases changed. It could also identify language considered offensive, insensitive or inappropriate and replace it with alternative wording. The patent suggests that the system may use an internet search to find suitable replacement words.

These changes could be based on predefined instructions or individual user profiles, giving viewers the option to customise how certain words and expressions are presented in a live feed.

Microsoft Patent Could Store Original Audio And Video

The proposed technology could be used for different types of live content, including lectures, business presentations, award ceremonies, television shows, cooking programmes and live sports.

However, changing a speaker's words and appearance also brings up the question of how the original footage would be preserved. The patent describes storing the unmodified audio and video as “ground truth”. This would provide a reference against which the edited version could be checked.

The system could allow the same speech to appear differently to different viewers, depending on the changes applied to their feeds. While the technology could help with translation and speech correction, it could also change a speaker's expressions and wording.

Microsoft has not confirmed whether it plans to develop the technology into a product. As with other patent filings, it remains unclear whether the proposed system will ever become available to consumers.