Let's Make Robots!

Using gestures and voice commands to control the robot

Couple of months ago we decide to participate in Intel's perceptual computing challenge (http://perceptualchallenge.intel.com). The idea was to extend driver cockpit application to control Veterobot (http://veterobot.org) with gestures and voice recognition. We are using detected thumb position in 3D space. Position along horizontal axis is used as a steering signal and distance to the camera (depth) to control acceleration (forward and backward).

It was a fun exercise but some more tuning is still required to achieve smooth control experience. So it is work in progress.

Edit: somehow, I can not see the attached video in the blog post, so here is the link: http://www.youtube.com/watch?v=5AkzF8kIeC0

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I can imagine what you mean. But gesture does not have to with thumb. Why not train the robot to process gestures like arms up, arms to the side, one arm to the side...

That would be more recognizable i guess since arms are bigger than thumbs and the change of the picture before the background is also bigger.

Anyway, sometimes you try something and then you realize that there is a better way but what you have learned might help you later. So it's not wasted time :-)

> Why not train the robot to process gestures like arms up, arms to the side, one arm to the side...

Wait, I probably did not understand what you mean and was answering too fast. Do you actually mean to do recognition on the robot itself, i.e. using on-board computer? In my case, gesture recognition was made on the PC, i.e. where the driver console application was running.

If yes, then I think it is indeed coulde be useful in case where there is no second computer with control software to drive the robot. However, it will require the driver to be in the field of view of the camera, which is quite a limitation. So here is also plenty of problems to think about...

Well, yes. For an application like field operation the robot could do some tasks on specific gestures. I could imagine to gesture the robot from a remote location (but still in camera sight) to bring me my toolbox or in private sector gesture him to bring me a beer while I am lying in the garden.

Haha, a note like that in your resume is quite useful :-)

> But gesture does not have to with thumb. Why not train the robot to process gestures like
> arms up, arms to the side, one arm to the side...

I am not sure that it will be improvement compared to the mouse or or joystick. Also, based on my experience with Intel's perceptual SDK, the recognition rate even for the thumb is quite good. So it is less the question of recognition, but more of general usability of gestures to control the robot.

> Anyway, sometimes you try something and then you realize that there is a better way but
> what you have learned might help you later. So it's not wasted time :-)

I am totally agree with you here! At the very least, now I can mention gesture recognition technology in my resume which is already positive result of these efforts :-)

A year or so ago Google's April Fool's day video was a gesture-based email UI. It was hilarious, but I think it demonstrates what you learned-that maybe it's really a specialized application. It might be useful for people with fine-motor control disabilities but for the rest of us a joystick or thumb pad probably makes more sense.

As I understand, the trend goes towards building this kind of technology into mobile phones and tablets as an alternative to the touch interface. In this case it might be useful. At least for simple actions like starting applications, selecting browser bookmarks, scrolling, etc.

Well, that's a interesting project. The gesture recognition is great. очень хорошо :-)

Thanks Lumi for the positive feedback!

Gesture recognition technology is indeed interesting area, however I am not sure whether it makes sense for this kind of robot control. The reason is - after about 30 minutes of intensive testing my shoulders were really tired. It is necessary to keep hands straight in front of the camera and it is not very convenient during the long period of time. And honestly, I can achieve pretty much the same effect with mouse.

So for me it was a nice exercise to learn and play with promising Intel's technology but I am not convinced yet that it is well suitable for this particular application scenario. Intel's perceptual SDK also provides rather simple way to track the head movement and I was thinking about using it instead of hands tracking. Than I realize that if I want, for example, turn left or right, then I need to rotate the head, but at this moment I can not see the monitor anymore which is also not perfect for control :-) . So there is still room for improvements.

> очень хорошо

Я старался :-)