Let's Make Robots!

Realtime Lipsyncing for Animatronics


Lipsyncing is the process of making an animatronic costume or model mimic the actions of speaking. I use the term mimic as it is currently impossible to make an artifical tougue or larynx that will duplicate speech. So we have to use a speaker with a recorded voice and have an animatronics track to make the movements.

Real Time Lipsyncing is where you do not have the luxury of being able to record tracks and add the lipmovements. It has to be done on the fly and often whilst interacting ad-hoc with an audience.

This is my concept of how it can be done usingoff the shelf technologoes and mininmal computing power.

First off I will be making and wearing a lipshield. This is a flexable silicon mask that is glued to my lips. It will have 8 embedded IR SMD leds set up as 2 circuits (A-B-C-D) and (E-F-G-H). A WII IR Camera will be positioned so that it can see all LED's (A fisheye lens will be needed)

The microcontroller (Picaxe or Arduino) will strob the 2 sets of LEDS and thus beable to read the 8 point in 2 sets of 4 co-ords.

Point A is important as it will indicate how much the mouth is opened. The other points can be used to adjust servo's around the characters lips so to simulate the speech pattern.

Where the character has a tongue that can be extended we can use a IR Emitter/Detector module focused at the blue circle. Whilst the other LEDS are turn off this LED can be illuminated and so via reflection from the tongues surface give an approximation of how far the tougue is extended (more refection more tongue). Also by using Point A it can work out if the mouth is opened and thus stop false reading.

All parts are off the shelf and I will be using Chris the Carpenters WII Camera module and possibly a PicAxe 28x2. The IR detector is here http://littlebirdelectronics.com/products/qre1113-line-sensor-breakout-analog

Many thanks for following my ramblings

Dave

p.s. - The animatronic jaw can be seen here http://www.marcwolf.org/gallery/default.aspx?aid=4

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Please keep us informed. :)

Now to figure out how to make it work for those of us that prefer facial hair. :)

LOL. nice hint :-)

Low tech solutions.. RAZOR....

But seriously - the completed latex parts will be basically around my lips. I wanted to mate the working area larger so I could experiment with the placement of the LED's.

A simpler solution to what I am doing would be just having sequins that you glue onto one's lips and use a IR source to reflect from them. But the concept will still be the same and would work for those who  wear beards.

 

Dave

Low tech solutions.. RAZOR....

But seriously - the completed latex parts will be basically around my lips. I wanted to mate the working area larger so I could experiment with the placement of the LED's.

A simpler solution to what I am doing would be just having sequins that you glue onto one's lips and use a IR source to reflect from them. But the concept will still be the same and would work for those who  wear beards.

 

Dave

Hmm, bird's avatar does look like it can use a Razor...

Easy there youngin. :P

Hi Folks

These are a few pictures about me making the lip moulds (Not for the squeemish)

NO NO.. It's not what you think!!!. I have filled this small container with dental alginate and pressed it to my face to get an impression of my lips and chin.
Making the Alginate mouth mould

Here is the finished negative. Alginate is great as it can capture the finest details like fingerprints.Finshed Mould

Now to fill it with plaster of paris to make the positive

And finally the finished item(s). I made to positive one with the lips seperated and one without. The original mold is at the top.

Finished

Now the next stage is to paint on some liquid latex and build up the lips. I'll also be embedding the LED's into the lips so that they will follow my movements as best as possible.

Take Care
Dave