Let's Make Robots!

Anna is one year old now.  She is learning quickly of late, and evolving into primarily a learning social creature and aggregator of web services.  I wanted to document where she is at her one year birthday.  I need to create some updated design diagrams.

Capabilities Achieved in Year #1

1)      Thermal Array Vision and Trackingused to keep face pointed on people it is talking to, or cats it is playing with.

2)      Visual Tracking - OpenCV to search for or lock onto color shapes that fit particular criteria

3)      Learns by Listening and Asking Questions - Learns from a variety of generic sentence structures, like "Heineken is a lager", "A lager is a beer", "I like Heineken", "Olive Garden serves Heineken"

4)      Answers QuestionsExamples:  "What beers do I like?", "Who serves Heineken?", "What does Olive Garden serve?"

5)      Understands ConceptsExamples:  is a, has a, can, can’t, synonym, antonym, located in, next to, associate of, comes from, like, favorite, bigger, smaller, faster, heavier, more famous, richer, made of, won, born in, attribute of, serve, dating, sell, etc.  Understands when concepts are similar to or opposite to one another. 

6)      Makes Smalltalk & Reacts to Common ExpressionsMany human expressions mean the same thing.  Example:  “Hows it going?”, “Whats up?”, “What is going on?”, “Whats new?”   A robot needs many different reactions to humans to keep it interesting.  Example: “Not much, just keeping it real”, “Not much, what’s new with you?”

7)      Evaluates the Appropriateness of Topics and Questions Before Asking Them - Example: Don’t ask someone : “Who is playing on Monday Night Football tonight?” unless it is football season, Monday, and the person is interested in football.  Also, don’t ask a kid something that is not age appropriate, and vice versa, don’t ask an adult how they like the third grade.  Don’t ask a male about his gynecologist.  This is a key piece of a robot not being an idiot.

8)      Understands Personal Relationships - it learns how different people you know are related to you, friends, family, cousins, in-laws.  Examples: “Jane is my sister”, “Mark is my friend”, “Joe is my boss”, “Dave is Mark’s Dad”   It can answer questions like “Who are my in-laws?”, “Who are my siblings?”, “Who are Mark’s parents?”

9)       Personal Info - it learns about both you and people you know, what you like, hate, answers to any questions it ever asked you in the past.  Example:  “My wife likes Nirvana” – in this AI had to determine who “my wife” is.  It can then answer questions like “What bands does my wife like?”, as long as it already knew “Nirvana is a band”

10)      Pronouns – it understands the use of some pronouns in conversation.  Example:  If I had just said something about my mother, I could ask “What music does she like?”

11)   Opinions – the bot can remember your opinions on many things, and has its own opinions and can compare/contrast them to add color to a conversation.  Example:  If I said, “My favorite college football team is the Florida State Seminoles” it might say “That is my favorite as well”, or “My favorite is the Alabama Crimson Tide”, or “You are the first person I have met who said that”

12)    Emotions - robot has 10 simulated emotions and is beginning to estimate emotional state of speaker

13)    Motivations - robot has its own motives that take control of bot when it is autonomous, I keep this turned off most of the time.  Examples:  TalkingMotive, CuriosityMotive, MovementMotive

14)   Facial Expressions - Eyes, Eyelids, pupils, and mouth move according to what robot sees, feels, and light conditions

15)   Weather and Weather Opinions - uses web service for data, programming for opinions.  Example:  If the weather is freezing out and you asked the robot “How do you like this weather?”, it might say “Way too cold to go outside today.”

16)   News - uses Feedzilla, Faroo, and NYTimes web services.  Example:  say something like  "Read news about robotics", and "Next" to move on.

17)   TV & Movie Trivia - plot, actors, writers, directors, ratings, length,  uses web service.  Example:  you can ask “What it the plot of Blade Runner?”, “Who starred in The Godfather?”

18)   Web Search - uses Faroo web service.  Example:  say "Search web for Ukraine Invasion"

19)   People - uses Wikipedia web service.  Example:  "Who is Tom Cruise?", “Who is Albert Einstein?”, “List Scientists”, “Is Clint Eastwood a director?”, “What is the current  team of Peyton Manning?”, “What is the weight of Tom Brady?”

20)   Trending Topics - uses Faroo web service.  Example:  say something like "What topics are trending?", you can then get related articles.

21)   Geography - mostly learned, also uses Wikipedia.  Watch the video!  Examples: "What is the second largest city in Florida?",  "What is the population of London?", “Where is India?”, “What is next to Germany?”, “What is Russia known for?”, “What is the state motto of California?”, “What is the state gemstone of Alabama?”, “List Islamic countries”

22)   History - only knows what it hears, not using web yet.  Mostly info about when various wars started, ended, who won.  Robot would learn from:  "The vietnam war started in 1965" and be able to tell you later.

23)   Science & NatureExamples:  "How do I calculate amperes?", "What is Newtons third law of motion?", "Who invented the transistor?", "What is the atomic number of Gold?", “What is water made of?”, “How many moons does Mars have?”, “Can penguins fly?”, “How many bones does a person have?”

24)   Empathy - it has limited abilities to recognize when good or bad things happen to people close to you and show empathy.  Major upgrades to this have been in the works.  Example:  If I said, "My mother went to the emergency room”, the bot might say “Oh my goodness, I am so sorry about your mother.”

25)   2 Dictionaries– Special thanks to Princeton and WordNet for the first one, the other is built from its learning and changes constantly as new proper names and phrases are encountered.  You can ask for definitions and other aspects about this 200,000 word and phrase database.  You can add new words and phrases simply by using them, the AI will save them and learn what they mean to some degree by how you use them, like “Rolling Rock is a beer”, AI doesn’t need anything more, nor would a person.

26)   Math and Spelling- after all the other stuff, this was child's play.  She can do all the standard stuff you can find on most calculators.

27)   The AI is Multi-Robot and Multi-User - It can be used by multiple robots and multiple people at the same time, and tracks location of all bots/people.  Alos, A given Robot can be conversed with by multiple people at the same time through an android app

29)   Text Messaging - A robot can send texts on your behalf to people you know, like "Tell my wife I love her."  - uses Twilio Web Service

30)   Obstacle Avoidance - 9 sonars, Force Field Algorithm, Tilt Sensors, and down facing IR cliff sensor keep the bot out of trouble

31)   Missions - robot can run missions (series of commands) maintained through a windows app

32)   Telepresence - robot sends video back to server, no audio yet, robot can be asked to take pictures as well.   Needs improvement, too much lag.

33)   Control Mechanisms - Can be controlled verbally, through a phone, tablet, web, or windows app.  My favorite is verbal.

34)   GPS and Compass Navigation – It’s in the code but I don’t use it much, hoping to get my Wild Thumper version of this bot built by summer.  This bot isn’t that good in tall grass.

36)   OCR - Ability to do some visual reading of words off of walls and cards – uses Tesseract OCR libraries

37)   Localization - through Recognizing Words on Walls with OCR – I don’t use this anymore, not very practical

38)   Lasers - I almost forgot, the bot can track and hit a cat with lasers, or colored objects.  It can scan a room and shoot everything in the room of a particular color within 180 degrees either by size or some other priority.

39)  I know I singled out Geography, Science, Weather etc as topics, mostly because they also use web services.  The AI doesn't really care what it learns,  it has learned and will learn about anything you are willing to tell it in simple sentences it can understand.  It can tell you how many faucets are on a sink, or where you can get a taco or buy a miter saw.

Goals for Year #2

1)      More chat skills – I fear this will be never ending

2)      More Hard Knowledge - we can always learn more

3)      More web services – takes me about a day to integrate a new web service

4)      Face Tracking - know any good code/APIs for this?

5)      Facial Recognition - Know any good free APIs for this?

6)      Arms - I like to get some simple small arms on just to be more expressive, but will have to redesign and rebuild the sonar arrays to fit them in.

7)      Empathy over time - I'd like the bot to visit good/bad events and ask about them at appropriate points in time later. Things like "How is your mother's heart doing since we last talked?"  I have done a lot of prep for this, but it is a tough one

8)      More Inquisitiveness and Initiative - when should the bot listen and when should it drive the conversation.  I have tried it both ways, now the trick is to find a balance.

9)      Changeover to Newer Phone

10)   Go Open Microphone - right now I have to press the front of my phone or touch the face of the bot to get it to listen, I’d rather it just listen constantly.  I think its doable on the newer phones.

11)   Get family, friends, and associates using AI on their phones as common information tool about the world and each other.

12)   Autonomous Learning - it can get info from the wikipedia, web, news, web pages, but doesn't yet learn from them.   How do you build learning from the chaos that is the average web page?  Listening was so much easier, and that wasn’t easy.


·         Arduino Mega ADK (in the back of the body)

·         Arduino Uno (in the head)

·         Motorola Bionic Android Phone

·         Windows PC with 6 Cores (running Web Service, AI, and Databases, this PC calls the other web services)

·         Third Party Web Services - adding new ones whenever I find anything useful


·         I would love to hear any suggestions anyone out there might have.  I am constantly looking for and reevaluating the question “What next?”

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
KIDBOT's picture

I am now working on a system of similair design, which is just a huge advancement over C.A.R.L. I am currently working on the "structure" fo the though process (which your work has helped greatly with), increasing my python skills and the hardware component. I am currently trying to get my hands on as much as I can regarding AIs and mental processes. I just hope that both of us can develop the "AI of our dreams". :D  

If you are trying to learn about AI, check out some free online classes through Georgia Tech.  They have several on AI and the price is right.  They are going to have some machine learning classes that will go live in April which would be especially applicable.


I am doing


which is taught by Sebastian Thrun who architected the Google Car project.  I am halfway through it, and I think it is excellent although to say I understand it all would be a stretch.



mtriplett's picture

Thanks for the recommendation.  I watched all the trailers and read the info.  Very interesting, defintely something I need to do.  Sebastian Thrun is a hero of mine.  He was in the Great Robot Race, the documentary on the Darpa robot car race (watch it on Netflix).  Within 10 secs of him speaking, I said to my wife "My money is totally on this guy to win."

On the classes, I think I might start with the machine learning ones.  I really hope I can make the time.  What would probably happen though is I would get 15 minutes in, I'd get an idea, and then want to drop everything and work on it.  I'll let you know if I sign up.  Good luck creating your own driving car!



JerZ's picture

"The Great Robot Race" was on PBS, so if anyone has an hour to kill, this is a good way to do it.

via YouTube:

The Great Robot Race

mtriplett's picture

Hope you have a lot of spare time.  Its mega rewarding to me.  I'm toying with setting up the AI as a web service that other LMR hobbyists can use by the end of year.  I built mine from the ground up without benefit of doing a lot of reading though, spotting patterns and iterating as I went along.  It probably doesn't measure up to academic theory at all, but to me it is practical and easy to evolve.

I'm now redesigning and iterating again.  Like you I'm trying to spend more time reading about what's going on out there by bigger and smarter teams.  I keep hoping for an easy API to IBMs Watson.  I ran into "The Open Cognition Project" the other day, interesting and intelligent but complex.  I did see some parallels with my own work, although I'm using some other techniques I have yet to run into elsewhere.  I am trying to learn what I can from them.  I'm thinking of this as 1 year into a 15+ year effort. 

If you ever want to trade ideas/designs let me know.  Good luck with your endeavor.



KIDBOT's picture

Honestly, every spare moment I find myself reading about the human mind, researching proir attempts at strong AIs, and trying to compile that into a "logic".

I cannot believe how much work you have gotten done on her in the past year.  It is incredible what you have done.  Wow.



mtriplett's picture

Thanks Bill.  In one of the videos Anna said "You don't get out much do you."  She was onto something.



I don't get out much either.  And I have MUCH less to show for that time over the past year than you do!  My 2 bots so far are toys by comparison.  I haven't posted anything I have done since they don't really do anything interesting or innovative. 

I noticed in your other posts,you might open source this.  If you do decide to open source this, I could work with you on this.  I am a C# middleware guy by trade, and I thought you had posted elsewhere these are all web services so assume WCF SOAP services for transport.  You might think about the WebApi and making these REST services to lower data overhead - can make it JSON and Java should have a serializer to deal with that on the Android side.  If not, it is a very simple protocol to deconstruct. 

Honestly, you might not want to go that route.  You have something very close to sellable to a company if you want to quit your day job.  Some more spit and polish on your girl (it is only time or less time and more money to get there), and it wouldn't be hard to imagine many, many areas where she could be used. 

Or just keep playing in your spare time and have fun with her.  You have done some incredible stuff here which can only get better with time.  A year from now, she would be that much more believable...




mtriplett's picture

I agree with your input...I'm doing a re-organization now after starting an "Intro to AI" class online, taking a step back and reassessing my code from a software architecture standpoint.  I have a lot of ideas for simplifying the flow and making it more flexible and powerful at the same time.  I'm also renaming things to be more in-line with academic terminology on AI...like my Responders are now "Agents".  I'm also preparing to move to having a given request be able to run on multiple threads, since each Agent is mostly independent of all other agents.  

You mentioned JSON...I've been using a lot of JSON web services, so I can see that it would be nice to serve up my own web service as JSON or JSONP.

I'm trying to move the code in a direction where multiple bots could use the service while the owners of those bots would be able to customize which agents would run for which bots (and how they prioritize) through a web or windows app, or on the fly at runtime (the bot changing its own agents).  Under this model, an agent could be any chunk of code that people want to run on the server that implements a very simple interface  (not necessarily a language related agent).   An agent could also be a proxy for something running on someone else's server.  The main thread would basically just run whatever agents are configured for a given bot, wait for them all to finish (think multiple threads) and arbitrate on what behaviors win and get sent back to bot.  I'd probably give people the first 60-100 agents to play with.  This would basically encompass the verbal skills, learning, emotions, motivations, and of course the existing knowledge base.  I'm hoping to come up with a mechanism for people to create, test, and share new agents.  Any bot with web connectivity could then access the service.

I think if I can find a half dozen people with bots that are excited to give it a try, it could be quite a fun project and a lot of new ideas would be generated along the way.