Let's Make Robots!

A Friendly Overlord

A Python Module that Tracks and Controls a Robot using OpenCV

I've been working on this one in silence for a bit.  

Awhile back it hit me, before I started growing my Overlord project in complexity I wanted to refine it for ease-of-use. Therefore, I began translating my Overlord project into a Python module I could build off.

I figure, this would make it easier for anyone to use. This includes myself, I've not forgotten my identity as a hack, nor will anyone who pops the hood on this module :)  

But, at its core, there are few essential inputs:

  1. Color to track.
  2. Compass reading.

So, I spent some time translating the code into a callable module.  This experiment was mainly for my own use, yet I knew it'd grow healthier if I had LMR's feedback, elder or noob.

When I started I actually planned (gasp) out what would make this code more user friendly.  I didn't think long; the two things that have taken the most time tweaking to get this code useful are:

  1. Adjusting the compass heading.
  2. Selecting the color to track.

To address the first issue, I developed a "auto-compass calibration function."

def mapper(x, in_min, in_max, out_min, out_max):
    #This will map numbers onto others.
    return ((x-in_min)*(out_max -out_min)/(in_max - in_min) + out_min)

def compass(headingDegrees):
    global compassInitFlag
    global initialRawHeading
    global intRx

    #This sets the first compass reading to our 0*.
    if compassInitFlag == False:
       initialRawHeading = headingDegrees
       compassInitFlag = True
       print initialRawHeading

    #This is the function that actually maps offsets the compass reading.
    global intialRawHeading
    if headingDegrees >= initialRawHeading:
        adjHeading = mapper(headingDegrees, initialRawHeading, 360, 0, (360-initialRawHeading))
    elif headingDegrees <= initialRawHeading:
        adjHeading = mapper(headingDegrees, 0, (initialRawHeading-1),(360-initialRawHeading), 360)
    #Here, our compass reading is loaded into intRx
    intRx = adjHeading


Basically, this function takes the very first compass reading and adjusts all other readings.  So, all you have to do is put your robot in the direction you want it to consider "North," start your code, and this function will convert all other readings.


The second issue took me a little longer to deal with: easy color selection.  In short, I rewrote most of the color detection parts of the code to take advantage of the OpenCV's CamShift algorithm.  This function is more resilient to lighting changes or other near color objects, but it is also more CPU intensive.  At some point, I'll probably go back and write a variant that sticks with the old largest-target-color-mass method.  

Ok, what this means for the user?  When the code starts you select the color you'd like by left-click and dragging a selection box over an area.  The mean color of the selected area will be tracked and this will also start the rest of the code.

What does Friendly Overlord give you?

Well, a lot.  And when I finish writing the damn thing, more than alot.

Here's a list, and only one bit is untrue.

  1. It tracks your robot, providing its x and y relative to your webcam.
  2. It will provide a target coordinates, which I'll later make addressable in case someone wants to do something cool, rather than have their robot drive around and catch virtual dots. Lame.
  3. It will take the compass reading you provide, translate it to a heading relative to the camera, then, it will send commands to your robot telling it to turn until it is in alignment, then move towards the target.
  4. Make you a cuppa (CP, DanM, did I use that right?)
  5. It will allow you to tweak pretty much any element of the code (e.g., overlord.targetProximity = 5)

What does it not do?

  1. Take care of your serial data.  You're own your on, bud.
  2. Write your robot uC code for you.
  3. Provide you with your robot's heading (though, when I delve into two-color detection this could be done with two-dots on your bot.  But really, it'd be easier and near cheaper to get an HMC5883L).

Alright, so let's talk code.  How little code does it take to use it?

import serial
from time import sleep
import threading
import overlord

#Initialize Overlord variables.

#Open COM port to tether the bot.
ser = serial.Serial('COM34', 9600)

def OpenCV():
    #Execute the Overlord.

def rx():
        # Read the newest output from the Arduino
        if ser.readline() != "":
            rx = ser.readline()
            rx = rx[:3]
            rx = rx.strip()
            rx = rx.replace(".", "")        
            #Here, you pass Overlord your raw compass data.  

def motorTimer():
        #This is for threading out the motor timer.  Allowing for control
        #over the motor burst duration.  There has to be both, something to write and
        #the motors can't be busy.
        if overlord.tranx_ready == True and overlord.motorBusy == False:
            ser.flushOutput() #Clear the buffer?
            overlord.motorBusy = True
            overlord.tranx_ready = False
        if overlord.motorBusy == True:
            sleep(.2) #Sets the motor burst duration.
            sleep(.3) #Sets time inbetween motor bursts.
            overlord.motorBusy = False

#Threads OpenCV stuff.        
OpenCV = threading.Thread(target=OpenCV)

#Threads the serial functions.
rx = threading.Thread(target=rx)

#Threads the motor functions.
motorTimer = threading.Thread(target=motorTimer)


This is fully functional code.  You'll notice that really, only about 10 lines get Friendly Overlord going, the rest handle Serial functions and motor firing.  Be warned, the motor firing code will change, since it is written how I like it right now, eventually will be designed to be as flexible as possible.


  1. overlord.dVariables() #Sets the Friendly Overlord variables.
  2. overlord.otracker() # The module's heart.  Handles color tracking, angle calculation, etc.
  3. overlord.compass(x) # You pass it an compass heading as an integer in degrees (0-360) and it does the rest.
  4. overlord.tranx_ready # Simple flag to indicate last bit of serial data has be sent.
  5. overlord.tranx # Variable that contains the serial command to be sent to the robot.
  6. overlord.motorBusy # Flag to indicate if the robot is still in the middle of a movement.

That's about it.  In the module? 399 lines of code, or so.  Still relatively small for a program but not something I want to wade through without a damned good reason.

Ok.  So, where am I going with this?

Hell if I know.  I want to make it as versatile as possible.  Eventually, I'd like to be tracking nth number of robots.  I envision a swarm of Yahmez' Baby bots flying all over the place, Friendly Overlord tracking them, and communicating with them via IR.

But in the more immediate future, I'd like to make every variable tweakable.  Especially, variables useful to others.  For instance, the overlord.tX and overlord.tY are currently controlled by the module.  They are simply randomized numbers.  But, I'll make a flag in the next two days to take control of them from your own code.  You can decide where you'd like your robot to go.  Whether it be to your mouse pointer (overlord.targetY = overlord.mouseY) or a complex set of way-points to lead him through a maze.  Really, I'll probably code around the feedback I get.

Now, some obligatory stuff.

Here are some of the current variables addressable from your program:

#How close to does the robot need to be? Greater is less accurate.
#Defaults to 5.
overlord.targetProximity = 5

#GUI X, Y 
#Defaults to 0, 0
overlord.guiX = 440
overlord.guiY = 320

#Random target constraint; so target doesn't get placed too far from center.
#Defaults to 1, 640, 1, 480
overlord.targetLeftLimit = 20
overlord.targetRightLimit = 400
overlord.targetBottomLimit = 320
overlord.targetTopLimit = 20


But I'd like to make every variable needed by the user available.

Ok.  So, here's what I need: Someone to use it and provide feedback.  I'm getting too close to it and bleary of thought.

I've thought of doing a few things to get some feedback:

  1. Setup a challenge (I've got some surplus).
  2. Offer to mail one person a month a setup (two Bluetooth PCBs and a cheap webcam).

Any suggestions?

I think I'll make a walkthrough video pretty soon (kinda miss making stupid videos) but I'm a little worn out right now.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Overlord is a very interesting program. I really like the idea because you'd only need one server per house that ran it and all of your mapping would be done.

I had a similar idea with my "troupe" (Groucho, Harpo, Chico, and Zeppo) where Groucho would be the main robot and handle a lot of the storage and mapping for the others. I hadn't generalized this to a stationary server like you and mtriplett. Thank you, sometimes I make my own blind spots.

It always hides so much of the work when py just lets you import an entire C program like line 4 there does. I went back and scanned the Overlord.c source when I saw that. It's a very useful setup, isn't it? It allows for a more objective-based rather than map reading system, which is probably more useful when things are in motion and you're trying to get a job done.

Agreed. And to think, I didn't like Python until Kariloy convinced me of its power.

Between you and I, Mr. Maxhirez, I feel kinda silly posting this without an application video.  But, I needed some sort of landmark.

Hope you're doing amazing.

For "field work" you might have to identify landmarks to localize the Overlord, but this would allow you to do neat things like have a quadcopter over the field, run the camera feed back to your Open-CV processor to maintain the drones on the ground...



I've thought of some pretty interesting applications; but, nothing fancy until I get multiple objects working.  

I've been reading through the literature on different CV algorithms implemented in OpenCV, I think it is going to be possible to track multiple objects of the same color in 2D using a Watershed and Nearest Neighbor algorithms, this should take care of merging or splitting.  In 3D it's a little trickier, since you have to worry about occlusion.  If I were tracking a quadrocopter(s) I'd probably do a mix of sensors; adding either an altimeter or sonar on its butt.

I found a much better explanation of Watershed.

Thanks for the pointer, Ladvien. That indeed was an excellent explanation of Wateshed.

In my scenario above, I think you want two tiers of tracking capable... The Copter(s) would use gps for rough location, then simply feed their video to a listener on the overlord.

The overlord would first pick a few landmark features in the feed to lock the copters position for a field frame.  then any land based drones in field of vision would be tracked by the overlord through the copter's video feed.

I know it's a stretch, but the possibilities...

Imagine a scenario where you've got a large field of inhospitable terrain that needs to be searched (mine field?)  the copters would be positioned like cell sites over the site, and would be capable of handing off drones as they pass from one field of vision to another.




Doh. Yes, I misread your reply completely.  I think understand now.

Huh. Never thought of that.  I always envisioned this huge pole in the middle of a football field or something.  A copter(s) makemuch more sense.

Counter proposal, the copter could sit on the back of a larger robot, then, when it needs extra sensor data fly up, pick an "anchor point," and guide the ground unit?  I'd probably use the ground unit as a "yardstick," since we would know its exact size.  Then, the CV could help navigate rough terrain by identifying objects and their distance to the ground unit.

Though, right now I'm mainly worried about getting the tools together, packaging them neatly and easy to use, and then let others invent application.  But I'll definitely respond to whatever application someone chooses.