Speech Driven Lip Synchronization

From CRAVEwiki

A research based project to automate the lip synching procedure for 3D animation. The project was initiated and was implemented with a 70% success rate.

The technology used for the main module was not optimal and therefore decided to be re initiated.

CRAVE is currently looking for a few dedicated members to work on this project.

The module two of this project was successfully tested and showed satisfying results but the module one needs a complete new approach.

Idea

An rough idea is to represent the sound/ speech wav file in a spectrogram form. An intelligent module capable of reading the spectrogram and recognizing the right phonemes is an requirement. Once the phonemes and their respective time of occurrence is recognized then accordingly shape keys corresponding to the appropriate phoneme will be placed on the timeline.

The second module is written in python integrating with Blender using its API.