Real Time Lip Syncing on the cheap

Lipsyncing is the process of making an animatronic costume or model mimic the actions of speaking. I use the term mimic as it is currently impossible to make an artifical tougue or larynx that will duplicate speech. So we have to use a speaker with a recorded voice and have an animatronics track to make the movements.

Real Time Lipsyncing is where you do not have the luxury of being able to record tracks and add the lipmovements. It has to be done on the fly and often whilst interacting ad-hoc with an audience.

This is my concept of how it can be done usingoff the shelf technologoes and mininmal computing power.

First off I will be making and wearing a lipshield. This is a flexable silicon mask that is glued to my lips. It will have 8 embedded IR SMD leds set up as 2 circuits (A-B-C-D) and (E-F-G-H). A WII IR Camera will be positioned so that it can see all LED's (A fisheye lens will be needed)

The microcontroller (Picaxe or Arduino) will strob the 2 sets of LEDS and thus beable to read the 8 point in 2 sets of 4 co-ords.

Point A is important as it will indicate how much the mouth is opened. The other points can be used to adjust servo's around the characters lips so to simulate the speech  pattern.

Where the character has a tongue that can be extended we can use a IR Emitter/Detector module focused at the blue circle. Whilst the other LEDS are turn off this LED can be illuminated and so via reflection from the tongues surface give an approximation of how far the tougue is extended (more refection more tongue). Also by using Point A it can work out if the mouth is opened and thus stop false reading.

All parts are off the shelf and I will be using Chris the Carpenters WII Camera module and possibly a PicAxe 28x2.  The IR detector is here

Many thanks for following my ramblings


p.s. - The animatronic jaw can be seen here



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I've moved this to the Projects as I will be posing a lot of steps in the development.



Lipshield, lipshield, lipshield....wanna see it ;-)

Well, you set high goals but I am thinking that it could work. Basically you just need to read the shields output and transfer it in a readable format to the servos moving the lips on your animatronics. This should not require much calculations.

I agree with NilsB, I want to see your lipshield.

I hear they are also handy when being kissed by female secret agents who wear poison on their lips.

Also know as Tranquility Lips in the series Alias. :)

It is my understanding that the Wii camera can only track 4 points at once.


I would have taken the promblem further in using the WII for the tounge sensor but it was difficult to find the intensity of the reflected IR  light

For the intensity of the reflected IR light you will be better off using a 5mm IR LED and Phototransistor similar to what I use with the IR compound eye:

He writes that he makes a time shifted sequence of four LEDs. That is ...

1. blink 4 LEDs
2. read the 4 values.
3. Blink the second 4 LEDs and
4. read the second 4 values.

This gives 8 values to make further calculations. That works.

Dave, I'm looking forward to see a picture of you wearing the lipshield. This may be one of the nerdiest devices we've seen so far ;-)