SeeStorm Products: SeeStorm Lip Sync SDK 2.0

SeeStorm Lip Sync SDK 2.0

SeeStorm Lip Sync technology enables 3D character's lips to be synchronized with voice input for automatically driving digital character speech. SeeStorm Lip Sync software analyzes the speech flow, generates natural lip motion corresponding to the speech and applies it to animate a 3D character face.

Lip Sync enables analyzing the speech flow (recorded or in real time) and generates natural lip motion corresponding to the speech in order to create a virtual character interface that creates fun and human touch. SeeStorm Lip Sync encompasses SeeStorm proprietary phoneme recognition and 3D phoneme visualization algorithms and allows extracting lip synchronization information from a speech signal to further apply it to a 3D character face.

Phoneme recognition is based on the Artificial Neural Networks technology (ANN). We use neural networks to classify the acoustic characteristics (coefficients obtained from analysis) in order to recognize phonemes. In addition, we use the energy variation to generate satisfactory co-articulation for the speech animation. It allows the real-time speech flow to be subdivided into several phonetic groups, corresponding to the basic lip motions during the speech and sufficient for the natural animation.

3D phoneme visualization provided by Face Mimic Modeling technology, as it uses 3D Model frame sequences to display lip motions and accompanying mimics, head, eyes, eyebrows all driven synchronously with the fluctuation, timing and nuances of the voice. To synchronize lip motions with voice in real-time, we analyze information in the decoded voice signal and display motion corresponding to the phoneme spoken. Using speech data to drive the model, which induces a probability distribution over the set of possible facial motions, produces animation. The output is a series of facial control parameters, suitable for driving many different kinds of animation ranging from video-realistic image warps to 3D cartoon characters. Realistic lip motions allow 3D character to be extremely natural and expressive.

SeeStorm Lip Sync is deployed in the US, Europe, Korea and Japan.