Let's Make Robots!

OMFG - It's Stereoscopic, Baby!

After about 8 months part-time development, after many VHDL-related headache, after lots of failures, after few successful moments, after anger, after surpises and finally with a big portion of luck I am today very proud to annouce the first working proof-of-concept for live stereoscopic vision using two CMOS camera modules, the Zeiss Cinemizer Plus video eyewear and of cause with my DIY side-by-side encoder.

What is the side-by-side encoder about? Well simply speaking it does nothing more than synchronising two video streams from the camera modules, horizontially shrinking them to 50% original width and rearranging them into a new video stream, where the images of both cameras appear on the left and right side of output video signal. This 3D-format is called side-by-side and can be watched e.g. with the Zeiss Cinemizer Plus or other suitable gear. If your have full control over your eye muscles your may even be able to enjoy 3D-vision with the crossed-eye method. If your want you can transmit such a video signal via standard RF-modules form FatShark or ImmersionRC. You don't need two video links for this purpose, one is enough as all the stereoscopic information is stored in a standard video frame. The video goggles takes this signal, splits it up in two halfs, descales it to original hoizontal size and displays it in the separate lcd or oled screens inside the goggles.

For a short demonstration I have mounted the two camera modules on my small robot like this. As I still do not possess a real workshop I prefer to work with balsa wood.

The small box on the left side encloses the SBS-encoder, which the robot has to carry on its back. Wireless video transmission is not yet done, but the RF-modules are already in my shelf and the wiring of all components is envisioned to be like this:

On the left the Zeiss Cinemizer video goggles with a FatShark receiver, on the right the camera box, the encoder and the video transmitter from ImmersionRC. The next photo shows a close-up of the encoder itself. With the exeption of the XuLA-200 daughter board it's all DIY.

Finally some remarks on the creation phase of this converter:

  1. In the begining of this project I have chosen the cmos camera modules rather arbitrarily, so I ended up with modules that output analog baseband video signals (CVBS). That's nice for a quick look on your TV set but a rather bad idea for what I have done here. Just imagine: the cmos camera module works in the digital domain and creates analog CVBS and the first thing I do is converting it back to digital domain BT.656 video stream. That's not very smart, as I recently leant that there are indeed ccd or cmos camera modules on the market with a digital BT.656 conforming output. Using two of such modules the encoder PCB you see above would shrink to half of the present size because all the components on the right hand side of the XuLA-200 FPGA are devoted to CVBS-to-BT.656 conversion (with two video channels).
  2. Don't underestimate the synchronisation issue of both video streams. This is really essential. You will need some kind of adaptive delay line in order to compensate for the (a priori unknown) delay between both streams. In my case I even had to use the external SDRAM resources of the XuLA-board since the internal SRAM of the FPGA is way too small. This was really a had fight to get the adaptive delay up and running.
  3. I was not able to use two camera modules out of the box, since the local XTAL jitter between both modules undermines any attempt to synchronize the video signals. See my other blog posts on this problem. I would estimate that a camera module with dedicated CLK input and digital video data output would not have this problem.

There is a video attached with some lousy live footage. Before you ask - I just grabbed some toys from my kids. If you have  3D-video glasses check it out or try the crossed-eye-trick. I think youtube is smart enough to offer more 3D-formats than just the side-by-side material I have uploaded.

So far for this moment. More is yet to come. Any comments are welcome.

Cheers, A.

 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Hi.

I have been working on a similar project myself.  However my goal is to provide a stereoscopic view whilst wearing a complex costume that I cannot see out of normally.

I took 2 video goggles and merged them by removing the video driver board from one and having each LCD driven by it's own board in the second.

It works very well and I even have a Propellor Backpack to overlay status's from the costume's animatronic drivers.

Take Care

Dave

Hi.

I have been working on a similar project myself.  However my goal is to provide a stereoscopic view whilst wearing a complex costume that I cannot see out of normally.

I took 2 video goggles and merged them by removing the video driver board from one and having each LCD driven by it's own board in the second.

It works very well and I even have a Propellor Backpack to overlay status's from the costume's animatronic drivers.

Take Care

Dave

Is it not practical to transmit two separate video streams on two different frequencies, and use two receivers at the goggles?

You can do it in any manner as your want. But IMHO using two transmitters and receivers goes along with a number of disadvantages:

  • Double equipment, double weight, double power consumption.
  • Waste of RF bandwidth. The transmitters and receivers I have chosen allow for only 8 different transmission channels. That's not really much.
  • More severe - the Zeiss Cinemizer has got only one CVBS input and expects the 3D-video data in the said side-by-side format.

I think this situation (as long as you want to stick with the Cinemizer) does not leave much opportunity for other approches. This may change when other video goggles become available (e.g. Cinemizer OLED, to be shown at the upcoming CeBIT fair).

I have been studying closely on 3D stuff lately and i came here, this thing is amazing. I heard you have to tag your video with a code to enable the viewers to access the 3D options. Now, for lately uploaded videos, you need to go to video info and select 3D or something. Easy way out, eye-crossing method. :)

I have been searching the web for some info on side-by-side encoding hardware without much success. Should be nice if you could share a block diagram of the encoder.

What is the aim with the stereoscopic project. Does this provide depth information? Can an estimate of distances be used by this method and does it improve object recognition? Or is this a project in forming 3d video for entertainment purposes.

Also, I am somewhat amazed, firstly that you did not find the digital cameras you would have preferred, but even more so at your ability to make a complex converter circuit to overcome this problem. I loved how you described the frustration part at the beginning. I think a lot of us here can relate to those feelings with projects that test us.

Thanks.

The aim of this work was to create a stereoscopic video stream that commercial video goggles can read. I was quite surprised to find out that there is AFAIK no stereoscopic set of cameras+goggles+Tx/Rx-gear on the market that just fits my needs. So I just took the challenge.

Depth estimation is IMHO more a matter of software algorithms for autonomous systems. It can be implemented, but at this time it is not my focus.