Let's Make Robots!

Accurate Indoor Localization using OpenCV, OCR

Just got this pieced together...Robot looks around, recognizes words written on walls and determines compass bearings to each.  Robot then determines what room it is in based on its memory of visual landmarks, and loads a map for the given room.  At this point, the robot calculates its location in the room to within a few inches.  Robot then knows where other features are within the room based on its map (Doors, Windows, Obstacles).  In the video, the robot announces the heading to the door.

I'm planning on using this for planning paths in a room and transitions through doorways from one room to another.

Hope you enjoy!  It was way fun to build.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

This looks very cool, I can't wait to see when you get it driving round, transitioning through doors. :)

Is your localisation done using just the landmark recognition and (I assume) triangulation? Or do you use the robot's sonar range finders as well?

You are correct, I'm only using the landmarks for localization indoors, GPS outdoors.  The sonars are used for obstacle detection and avoidance using a force field algorithm.  Using the sonars and and compass bearing goal, the bot can actually make its way into, through, and out of hallways now (in S curves, like it to be straight), even if the compass goal is way off of the direction of the hallway.

I could see starting to use the sonars for localization in smaller simple rooms like hallways.  I don't know how I'd go about doing the math in more cluttered rooms like I have here.  I'll have to study some more papers to get an understanding of that.

I'd like to incorporate more visual landmark recognition with OpenCV (not just the text), but I'd like the bot to do its own mapping/remembering/refining its map over time with multiple observations, using the text landmarks as reference points for accurately judging the position of the other non-text landmarks.  These capabilities are as of yet beyond my skillset.  I'll keep plugging away though, I find if I focus on a problem long enough, something can be figured out.

If you have any recommended reading or ideas, throw them my way.



Monte Carlo localization is my favourite algorithm to get started with, if a map is available, (although I've only implemented it in simulation myself). In fact, when I saw the ring of sonars on your robot, I thought that that is what you were doing. :) Also, your text landmark navigation system would work really well with MCL.

Apologies if mentioning MCL to you is like trying to teach you to suck eggs. However, if you're not familiar with it (or other people may find this useful) then the tutorial notes by Andrew Davison are a really good introduction to the topic, and they also have a link to the original paper by Dieter Fox.



Thanks much for your thoughts...very relevant to my situation and much appreciated.  I believe MCL would add a lot of value if

1) I can find some starter code to alter or if I can translate the academic work into working code.  (If you have any code to share, I'll send you a case of beer or some such.)

2)  If the sonar readings can be filtered well enough in real time...technically not hard but critical in my opinion.

I had read a few papers talking about the technique, never tried it.  While the concept is not over my head, the mathematical notation in academic papers often is.  On the sonars, I 've worked with them so much getting the sonar array going and have seen how wild they can be, I believe filtering the data appropriately before feeding it to MCL will have a huge impact on whether I get any useful output.

I believe I'd like to give it a try though.  Outdoors I think it will also be very necessary, as I've been wanting to see if I can get a bot to plan a path and move from my front yard to the back yard, with a house and landscaping blocking its way almost entirely.  I have some basic A* pathfinding I got working a while back, with a GPS map of the lot, house, etc., but MCL would really help I think as the bot approaches trees, house, fence, etc. as GPS is useless close in.

Thanks again,


Forget I mentioned source.  I should have googled it first, all kinds of stuff available.  Regards.

Glad you found something. The code I have (or possibly had) was a Python implementation for the Pyrobotics simulator, and it was probably a bit too adhoc to be of much use in the real world. Simulators are nowhere near as challenging as real life... ;)

Best of luck.


Gave me quite a bit to read and think about...brain is churning away.  So many options!