SMoRes (Searching while Modeling the Environment for Time-critical Rescues) [WEBSITE]

A First Response drone system designed for emergency responders to operate in perceptually degraded environments such as smoke and fire, while providing critical situational awareness through real-time mapping, human detection, and environmental reconstruction.

🚁 Overview

SMoRes is an advanced autonomous drone system that combines cutting-edge computer vision, simultaneous localization and mapping (SLAM), and multi-modal sensor fusion to enable safe and effective operations in challenging environments where visibility is severely compromised. The system is specifically designed for emergency response scenarios where traditional visual navigation fails.

Key Capabilities

• 🔥 Smoke & Fire Navigation: Operates reliably in environments with heavy smoke, fire, and poor visibility
• 👥 Human Detection & Tracking: Advanced thermal and RGB-based human detection and re-identification
• 🗺️ Real-time Mapping: Simultaneous mapping and reconstruction of degraded environments
• 🌡️ Multi-modal Sensing: Integration of thermal cameras, RGB cameras, IMU, and depth sensors
• 📡 Robust Communication: Reliable data streaming and communication in challenging conditions
• 🏥 First Responder Focused: Designed specifically for emergency response and rescue operations

🏗️ System Architecture

The SMoRes system consists of several integrated components:

Core Components

• SMoRes Basestation: Ground control station with visualization and mission control capabilities
• SMoRes Drone: Autonomous drone with multi-modal sensors and onboard processing
• HumanFlow: Advanced human detection, tracking, and re-identification system
• SLAM Workspace: Simultaneous localization and mapping algorithms
• Foxglove Extensions: Custom visualization tools and panels

Technology Stack

• ROS2 Humble: Robot Operating System for distributed computing
• Docker: Containerized deployment for consistent environments
• NVIDIA Jetson: Edge computing platform for onboard processing
• Thermal Imaging: Advanced thermal camera integration
• Computer Vision: YOLO-based detection and tracking algorithms
• SLAM: Real-time mapping and localization

📁 Repository Structure

smores_ws/
├── config/                    # Configuration files for all components
│   ├── foxglove/             # Foxglove visualization layouts
│   ├── logging/              # Data logging configurations
│   ├── pings/                # Network ping configurations
│   └── ros_domain/           # ROS domain configurations
├── foxglove_extensions/       # Custom Foxglove visualization tools
│   ├── bag_record_foxglove_panel/
│   ├── cloudini_foxglove/
│   └── foxglove-battery-extension/
├── HumanFlow_ws/             # Human detection and tracking system
├── launch/                   # Launch configurations for all components
│   ├── smores_basestation/   # Basestation launch files
│   ├── smores_drone_drivers/ # Drone driver configurations
│   └── smores_drone_isaac/   # Isaac simulation configurations
├── scripts/                  # Utility scripts for system management
├── smores_basestation_ws/    # Basestation ROS2 workspace
└── slam_ws/                  # SLAM algorithms workspace

🚀 Quick Start

Prerequisites

1. Docker Engine: Install Docker Engine using the official Docker documentation.

2. NVIDIA Container Toolkit:(Can be ignored for basestation without GPU) Install the NVIDIA Container Toolkit according to the official NVIDIA documentation. This also requires the CUDA Toolkit and NVIDIA Driver, which can be found here.

3. Foxglove : Install Foxglove using the official Foxglove documentation.

4. Other Dependencies: Install the following dependencies using apt:

   sudo apt-get update
   sudo apt-get install -y curl dpkg-dev git lsb-release openssh-server python3-pip rsync sshpass tmux tmuxp

Basestation Setup

1. Clone the repository:

   cd ~
   git clone git@github.com:castacks/smores_ws.git
   cd smores_ws
   git submodule update --init --recursive

2. Update environment variables: Update the SMORES_PATH and NDDS_QOS_PROFILES variables in "config/smores_basestation.env" based on the current root location of smores_ws code. eg:- "/home/user_name/smores_ws". Also update BASESTATION_IP to an available ip address on your system where foxglove can start bridge.

3. Set up environment variables:

   cd scripts
   source set_env_basestation.sh  # For basestation
   source ~/.bashrc

4. Setup docker containers:

   # Pull docker images
   cd $SMORES_PATH/launch/smores_basestation
   ./pull.sh
   
   # Build and launch the docker container
   docker compose up --build basestation-drivers -d #Build with GPU support
   docker compose up --build basestation-drivers-no-gpu -d #CPU only build

5. Access tmux terminals:

   # Enter the running docker container
   docker exec -it smores-basestation_ros2 bash #GPU build
   docker exec -it smores-basestation-no-gpu_ros2 bash #CPU only build
   
   # view running tmux containers
   tmux a

6. Foxglove setup: For foxglove panel layout can be found in "$SMORES_PATH/config/foxglove" and the code for custom extensions can be found in "$SMORES_PATH/foxglove_extensions"

🔧 Configuration

Environment Variables

Key environment variables that need to be configured:

• SMORES_PATH: Path to the SMoRes workspace
• BASESTATION_IP: IP address of the basestation
• ROBOT_NAME: Name identifier for the robot/drone
• ROS_DOMAIN_ID: ROS2 domain ID for network isolation

Sensor Configuration

The system supports multiple sensor configurations:

• Thermal Cameras: Dual thermal camera setup for stereo vision
• RGB Cameras: High-resolution color cameras for visual navigation
• IMU: High-precision inertial measurement unit
• Depth Sensors: RealSense cameras for depth perception

📊 Data Streaming & Visualization

Foxglove Integration

The system includes custom Foxglove extensions for advanced visualization:

• Bag Recording Panel: Automated data recording with configurable topics
• Cloudini Integration: Point cloud visualization and compression
• Battery Monitoring: Real-time battery status visualization

Data Topics

Key ROS2 topics for system monitoring:

• /thermal_left/preprocd_image - Left thermal camera feed
• /thermal_right/preprocd_image - Right thermal camera feed
• /drone/poi_detection - Human detection results
• /nvblox_node/pessimistic_static_esdf_pointcloud - SLAM point cloud

🛠️ Development

Adding Custom Components

1. ROS2 Packages: Add new packages to the appropriate workspace (smores_basestation_ws or smores_drone_drivers)
2. Launch Files: Create launch configurations in the launch/ directory
3. Foxglove Extensions: Develop custom panels in foxglove_extensions/

📈 Performance Monitoring

The system includes comprehensive health monitoring:

• System Health: CPU, memory, and network monitoring
• Algorithm Health: SLAM, detection, and tracking performance
• Sensor Health: Camera, IMU, and GPS status monitoring
• Communication Health: Network connectivity and data streaming

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• AirLab Stacks: Core development team
• CMU's DTC Team Chiron: Core development team
• Carnegie Mellon University: Research and development support
• First Responder Community: Requirements and feedback