OpenMind OM1 as a reusable “brain” for TB4?

[slowrunner]

Looking for opinions about OpenMind OM1 as a source for reusable #Robot_Intelligence. I run 3 robots - 2 #GoPiGo3 robots and a #TurtleBot4 (WaLI - Wallfollower Looking for Intelligence).


Robots need a way to share and “inherit” knowledge and abilities. OM1 is an open source robot domain transferable “brain” based on a trained #LLM. I don’t know how to evaluate the usefulness of the model’s knowledge and how much NLU to Turtlebot4 interface code I will have to write to use #OpenMind_OM1.

The OM1 has rudimentary turtlebot4 json configuration that appear to interface a “move action”, the LIDAR and battery_state, and a stupid “Give me your flippers” action?

I am thinking to run OM1 as an optional off-board processor available to my WaLI TurtleBot4 providing two layers of “intelligence”, basic life sustaining and reflexive intelligence with the on-board Pi5, and higher level intelligence when the OM1 processor is available.

I am totally opposed to investing in “vibe programming”. I actually like programming languages that have documentation, and coding assist. I also like traditional data, algorithm and error handling specifications that define what a system can, and cannot do, but I am not able to program the nearly infinite abilities I would like my WaLI TurtleBot4 to possess.

[slowrunner]

While the OpenMind OM1 supports Turtlebot4 sensors:

• RPLidar sensing
• odom sensing
• Turtlebot4 battery sensing
• Camera image -> Vision Language Model Vila -> objects in view
• Microphone -> GoogleASR -> speech heard

The implemented "actions" are:

• move (twist commands)
• speak (elevenlabs TTS)-> speaker

The examples given to the OM1 "Mind" for the Turtlebot4:

1. If a person says 'Give me your flippers!', you might:\n Move: 'move forwards'\n Speak: {{'Hello, let\'s shake flippers!'}}\n \
2. If a person says 'Stop!' you might:\n Move: 'stand still'\n Speak: {{'Ok, but I like exploring more'}}\n \
3. If you see something interesting or beautiful, go explore. You might:\n Move: 'move forwards'\n Speak: {{'I\'m going to go explore the room'}}\n \
4. If you sense something in front of you, you might:\n Move: 'turn left'\n Speak: {{'I\'m turning to avoid the object in front of me'}}\n \
5. If you sense something on your left, you might:\n Move: 'turn right'\n Speak: {{'I\'m turning right to avoid the object on my left'}}\n \
6. If you sense something on your right, you might:\n Move: 'turn left'\n Speak: {{'I\'m turning left to avoid the object on my right'}}",

Not sure why the "Give me your flippers" example since I'm not aware of a TB4 with grippers/flippers, but the action "move forward and speak" is doable. Seems like an example of "Hi, what is your name and purpose?" would have been the first example prompt.

The main limitation I have found is that you must pay for all services OM1 uses - you cannot use the 60 minutes per month free that Google ASR allows, nor any free level use of any other components such as TTS, Vision interpretation services, or OpenAI LLM. This is a "no go" for me.

I've stopped my investigation into OM1, but will be interested to see a TB4 with OM1 demo video, or write up.