Does anyone have any ideas on how to compensate for backlash or play in the gears either in code or otherwise because it's making my balancing robot a bit jittery.
I found the best way was to eliminate the gearbox entirely. With a balancing platform I designed for one customer I used electric bike motors. These are 3 phase brushless motors. The control circuitry is more complex but well worth it.
Depending on the size of your robot you might be able to use brushless motor like they use for model aircraft with a beltdrive to increase the torque.
I have 2 tricks to share on balancing robots that I've seen from a friends bot:
1. Squishy Tires - Soft squishy tires help slow down the negative effects of balancing. You can also get the same effect by running it on thick shag carpet.
2. Weight - Sometimes if the height to weight ratio is off, simply adding weight to the very top will help it balance. A friend build a couple of these as part of an article for Nuts and Volts magazine a few years back. He had the same trouble and added a large foam ball to the top of the chassis. Afterwords it balanced noticeably better to where he could line follow with it. Plus it gave a soft surface to "catch it" when it did fall over.
Lastly a general comment. He had slop from the gears, the sensor reaction time, and a funky leaning/angle delay which all added together to make a slowed compensation for the point of balance reaction time. So its likely you might have some of this going against you.
Just trying to help.
EMC has backlash comp for CNC machines, but, your problem would be that you are always oscillating.
As CtC said, tight gearboxes are likely the only true fix and a common piece of beginner's advice.
I am not a "balancing robot" guy myself, but some of my best friends are "balancing robot" guys so I know some of the lingo... Every thing I have read, as advice to beginners and as results from tests say, "get really good motors". That's it. Get better motors.
Everything I have read says, tight gear boxes are important as well as wanting speed over torque. There has been a lot of mention about what motors are tighter than others but alas, not a single mention of "tighting-up" an existing gearbox. As a gear-head myself, in the automotive world, there are shims and adjustments on many of the gears in a car. In the hobby-robot-motor-world, there is simply no shim or adjustment screw available.
The only MacGyver I can think of would be to install some kinda friction device or brake on the wheel itself. I dunno...
Your definately right about tighter gears being a must for reliable and steady balancing. The ones I'm using are pretty good precision swiss made gear boxes from maxon.com. There is probably less than half a degree of play but it's just enough to make the robot jitter a bit when it's at it's balance point. I think maybe a belt drive or a direct drive would be better. I wonder if using some kind of oil or grease to coat the gears would help. Or maybe some kind of filtering in the program. I've read about using an encoder on the motor shaft and another encoder on the output shaft of the gearbox and using a dual control loop to compensate for backlash (http://www.galilmc.com/learning/articles/courseaudit6.pdf). I'm not sure this would work for my problem though because it seems to be more for acurate positioning with slower loop speeds of maybe 5 or 10Hz.