Home System Architecture Reflection Milestone 1 Milestone 2

System Architecture

Overview

For this project, we split up into three subteams:
Machine Vision (Brooke and Lily): The machine vision subteam was focused on developing a machine vision pipeline that allows the chaser Neato to track and chase after the runner Neato.
Localization (Tchenzie and Zara): The localization subteam's goal was to create a system that allows Neatos to track their own poses within a known map and then publish those poses to each other. This allows the chaser Neato to follow the runner even when it can’t see it.
Multi-Neato and Integration (Brenna and Kelsey): The multi-Neato subteam worked on setting up the infrastructure for two Neatos to be controlled from one laptop. After achieving this they transitioned to working on integration between multi-Neato, localization, and machine vision subteams.

System Diagrams

architecture diagram
Initially we hoped to completely integrate our machine vision and localization code together with two Neatos. Unfortunately, an unsolved error on the localization side prevented us from completing this integration. The diagram above shows our initial architecture plan for the fully integrated project.
Both Neato1 and Neato2 have their own particle filter node since the locations of both on the world map are needed for the chase. Both particle filters send location data of their respective Neato to the chase_logic node which calculates what direction and speed is needed for Neato1 to follow Neato2, then sends those velocity instructions to Neato1. Neato2 gets its velocity instructions from teleop, so the user can play as Neato 2 in the game of tag.
architecture diagram
The diagram above shows our actual code architecture in our final project. This program switches controls between the Neatos when they tag. In each program, there is a dictionary attribute that stores which role is which neato. Upon a bump, the values swap. Blue arrows and text show the system when Neato1 is the chaser, and Neato2 is the escaper. Red shows the controls swapped. Refer to our README in our repo for a more in-depth breakdown on our logic (README File)

Machine Vision

Our original vision (ha!) for this subteam was using an algorithm like YOLO to detect another Neato and then SORT to track it. The image data would be enhanced with LIDAR, giving us the angle and distance of the other Neato. We quickly pivoted from YOLO after looking at 50 images we had taken from a Neato camera and seeing little to no contrast between another Neato and the space it’s in. We also realized that Neatos are too short to appear on LIDAR scans, and we’d need to modify them to be taller. Our solution to both of these problems was to cover the Neatos in bright pink sticky notes to increase visibility and height.
Once we did this, we realized that we could use a simple color detection algorithm to create bounding boxes rather than training a YOLO model. We tested this model on images of the pink Neato, and it worked extremely well. We then adapted the Neato Soccer code from Day 14 to move the robot towards the center of a bounding box. When we tested this, we found that the program worked exactly the way we expected. The next step was to add SORT tracking, which predicts where the bounding box will be next. We found that our first implementation of this performed poorly. We expect that this is due to our tracker not realizing that our Neato is moving as well, which changes the position of the bounding box even though the other Neato may not be moving. Due to the high performance of our program without using SORT, we decided to not use tracking and just update the bounding box with the current camera frame.

Localization

The localization subteam unfortunately wasn’t able to get the particle filter running on our multi-Neato setup, but these would be the steps to follow to run the multi-Neato localization.
The final localization system works by running a bash file “particle_filter.py”. This bash file executes 5 steps for both Neatos
Step 1: Initiate the Neato’s using the IP address.
The terminal asks for the last 2 to 3 digits of both Neatos. After the user inputs those digits it assigns it to the end of 192.168.16.{last digits here}. The bash file assigns both running Neatos to separate terminal tabs called “First Neato” and “Second Neato”.
Step 2: Initiate Multi-teleop for both Neatos.
Now that both Neatos are running, the bash file opens a new terminal tab called “Basic Teleop” which uses ros2 run to initiate the multi-teleop script created by the multi-neato and integration subteam.
Step 3: Special Ros2 Config
The bash file runs a line that sets up a special config for running RViz. We found that without this line of code, the particle filter does not work for the Neato.
Step 4: Particle Filter
The script attempts to run the particle filter for both Neatos at the same time using two separate terminal tabs called “Ros2 Particle Filter #{neato number}”. This line of code runs a built-in particle filter (AMCL) and launches the specified custom map we made in Milestone 1.
When running this line of code with multiple Neatos, we get the following message:
[amcl-2] [INFO] [1765823830.048684699] [neato2.amcl]: Message Filter dropping message: frame 'neato2base_laser_link' at time 1765823829.547 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
We do not know what causes this message because the particle filter works perfectly when setup for a single Neato (as shown in Milestone 2), and rviz seems to be displaying published data correctly, indicating that there is some problem with the AMCL node.
Steps we took to address this error (none of which worked, unfortunately):
  • Ensure all nodes are running on clock time, as opposed to simulation time. This is the most common cause of this error in general, but did not seem to be the cause in our case.
  • Increase the buffer size in the AMCL node’s message queue to account for latency in the Neato’s transform publishes
  • Consulting course assistants and our professor
Another step we could potentially take is re-publishing the transforms ourselves with appropriate timestamps, using a tool such as Chrony to sync the clocks between the Neato and our computers, or giving up on the AMCL particle filter entirely and using one of our own particle filters from the localization project earlier this year. Unfortunately we ran out of time to try any of these options, and had to cut our losses.
On the plus side, we did learn about static vs dynamic transforms, common ways to address timing synchronization issues in ROS2, and the structure of the launch files that we use to connect to the Neatos.

Multi-Neato and Integration

The goal of the Multi Neato subteam was to learn how to launch and control two Neatos on the same device, and to set up the publishers that would allow multiple files to interact with the Neatos. We began work early, and about halfway through the project we were able to dissolve to support the other two subteams.
We began by altering a simple, one Neato teleop code to control two Neatos. In order to successfully control two Neatos from one computer, each neato needs a specific namespace, which is used in all of the publishers to that Neato.
For example, in our teleop code we changed:
self.vel_pub = self.create_publisher(Twist, ‘cmd_vel', 10)
to:
self.vel_pub1 = self.create_publisher(Twist, 'namespace1\cmd_vel', 10)
self.vel_pub2 = self.create_publisher(Twist, ‘namespace2\cmd_vel', 10)
Similarly, To subscribe to information from a specific Neato, such as camera or lidar data, you need to use that Neato’s namespace as well.
Once we had successfully set up multi-Neato teleop, we began working with the localization and machine vision teams to adapt their code for multiple Neatos.
In our final code, the controls of the two Neatos swap when one tags the other. We achieved this by changing which namespace each control file (chase_logic.py and teleop.py) publishes to when the bump sensor is detected.