Skip to content

Chapter 8: Multimodal Perception and Intelligent Decision-Making

Visual perception is the process by which machines acquire environmental information through sensors and analyze and interpret images using computer vision technology. It encompasses tasks such as object detection and recognition, providing the system with environmental awareness capabilities. Obstacle avoidance decision-making relies on environmental information obtained through visual perception, employing environment modeling, path planning, and intelligent decision-making algorithms to formulate behavior strategies that avoid obstacles, prevent collisions, and achieve predefined objectives. The two are interdependent: visual perception provides environmental data for obstacle avoidance decision-making, while the latter executes actions based on this data. They play critical roles in fields such as autonomous driving and robot navigation, promoting the intelligent application and development of unmanned systems in complex environments.

8.1 Background and Theory

Multimodal perception and intelligent decision-making technologies are the core pillars for intelligent unmanned systems to achieve efficient collaboration, autonomous operation, and safety assurance, forming a closed-loop mechanism of "perception–cognition–action."

alt text

8.1.1 Multisource Information Fusion and Robust Perception

Traditional positioning and control schemes for unmanned systems often rely on single-sensor inputs such as GNSS satellite positioning. In highly interfered environments—including forests, urban canyons, over sea surfaces, or indoors—integrating multimodal sensor data (e.g., vision, LiDAR, IMU) has become an inevitable choice. Through techniques such as SLAM algorithms, robust, continuous, and dynamically updatable environmental models can be constructed.

8.1.2 Obstacle-Avoidance Path Planning and Intelligent Decision-Making

Obstacle avoidance planning relies on fused perception data. Combined with advanced methods including deep learning, reinforcement learning, behavior trees, graph search, and optimization algorithms, the system can autonomously assess obstacle risks, dynamically adjust speed and heading, and generate safe and efficient paths in real time.

8.2 Framework and Interfaces

The RflySim toolchain, combined with typical development cases, provides a detailed introduction to the supporting capabilities for intelligent perception and decision-making tasks, including sensor interfaces, data acquisition and processing workflows, and typical task algorithm architectures.

alt text

8.2.1 Image Acquisition in Virtual Environments

RflySim offers a high-fidelity virtual sensor simulation environment, supporting the generation of multimodal sensor data—including RGB vision, depth images, LiDAR, and IMU—providing realistic test data sources for visual perception algorithms.

8.2.2 Object Detection and Tracking

The platform supports algorithm validation for typical vision tasks, including object detection and tracking, path planning, and obstacle avoidance strategies. It provides standardized interface frameworks to help developers efficiently achieve the transition from simulation validation to real-machine deployment.

alt text

8.3 Showcase of Outstanding Cases

Five-UAV Visual Shared SLAM Hardware-in-the-Loop Simulation:

Simulation Algorithm Development and Validation:

8.4 Course-Linked Video Lectures

Public Lecture Replay for This Chapter (Session 7: Multimodal Perception and Intelligent Decision-Making):

8.5 Chapter Experiment Cases

The related verification experiments and guided cases for this chapter are located in the [Installation Directory]\RflySimAPIs\8.RflySimVision folder.

8.5.1 Interface Learning Experiments

Stored in the 8.RflySimVision\0.ApiExps folder, these experiments cover foundational platform interface tutorials and general introductions to various tools.

Experiment 1: Binocular Camera System Calibration

📝 Experiment Overview: Demonstrate binocular camera system calibration by altering the position and orientation of the chessboard in the RflySim simulation platform, and learn visual sensor configuration, image acquisition, and camera parameter tuning.

Experiment 2: ConfigJsonAPI Image Acquisition Interface Experiment

📝 Experiment Overview: Configure visual sensor parameters via the Python interface, acquire RflySim 3D images and control the drone in real-time, learning the usage of VisionCaptureApi and dynamic camera parameter adjustment.

Experiment 3: NX and Pixhawk6x Joint Hardware-in-the-Loop Simulation

📝 Experiment Overview: Implement joint hardware-in-the-loop simulation between NX and Pixhawk6x, controlling the aircraft to fly through a ring. This covers flight controller configuration, NX visualization configuration, workspace compilation, and control program execution.

Experiment 4: Visual Experiment Environment Preparation

📝 Experiment Overview: Summarizes the software environment, virtual machine environment, and hardware-in-the-loop preparation required before visual experiments, guiding users through Ubuntu, Anaconda, Pytorch, OpenCV, and network communication configuration to provide unified prerequisites for subsequent visual interface, distributed control, and algorithm experiments.

Experiment 5: MAVROS Python OFFBOARD Control Experiment

📝 Experiment Overview: Learn to use the MAVROS Python interface to switch the aircraft to offboard mode and arm it, achieving position control, and master ROS distributed simulation and WSL environment configuration.

Experiment 6: Getting Started with the RflySim Vision Interface

📝 Experiment Overview: Summarizes basic experiments related to the vision interface, helping users quickly become familiar with RflySim's image acquisition and control capabilities. It covers common interface topics such as images, depth, point clouds, IMU, timestamps, protocol transmission, and AirSim, providing interface-level preparation for basic vision control experiments and advanced algorithm experiments.

Experiment 7: UDP Direct Transmission PNG Compressed Image Distributed Simulation Experiment

📝 Experiment Overview: Learn to transmit images using UDP direct transmission with PNG compression, receive images on a remote Linux system or another Windows PC, and send back aircraft control commands, mastering distributed visual simulation experiment methods.

Experiment 8: MAVROS C++ OFFBOARD Control Experiment

📝 Experiment Overview: This experiment uses C++ to write a MAVROS program, switching the aircraft to Offboard mode and arming it, setting fixed coordinate points to achieve flight control. It covers workspace construction, simulation execution, and analysis of key knowledge points.

Experiment 9: Visual Box Hardware-in-the-Loop Simulation Ring-Crossing Experiment

📝 Experiment Overview: This experiment aims to implement visual box hardware-in-the-loop simulation, controlling the aircraft to fly through a ring. By configuring the visual box (with built-in NX and flight controller), hardware-in-the-loop simulation is completed via both serial port and Ethernet port methods, learning PX4 flight controller configuration, MAVLink communication, and visual sensor configuration.

Experiment 10: Automatic Generation of AI Training Dataset

📝 Experiment Overview: This experiment automatically generates image and point cloud AI training datasets through the RflySim3D simulation platform and Python interface, outputting them in VOC and KITTI formats respectively, learning visual sensor configuration and data acquisition.

Experiment 11: Mavros C++ Version Control Interface Experiment

📝 Experiment Overview: This experiment writes MAVROS control programs in C++, implementing drone control in both ROS1 and ROS2 versions within the RflySim simulation environment, and supporting distributed deployment, learning to use the MAVROS interface for flight control.

Experiment 12: Multi-Camera Image Acquisition Experiment

📝 Experiment Overview: Acquire RGB, grayscale, and depth images from three cameras via the Python interface, learning visual sensor configuration, image acquisition, and real-time camera parameter modification.

Experiment 13: PX4MavCtrlV4 Interface Control Demonstration

📝 Experiment Overview: Control the drone's flight attitude via the PX4MavCtrlV4.py interface, learning the use of position, velocity, attitude, and acceleration control commands, and mastering the software-in-the-loop simulation process.

Experiment 14: RflySim3D Object Position Acquisition

📝 Experiment Overview: Obtain the position and collision data of dynamically created objects in RflySim3D via the Python interface, learning the usage of the UE4CtrlAPI.

Experiment 15: Point Cloud Segmentation Experiment

📝 Experiment Overview: Obtain segmented point cloud data via the RflySim platform Python interface and display it in real-time, learning key knowledge points such as visual sensor configuration, point cloud display, and aircraft control.

Experiment 16: UDP Direct Transmission Point Cloud Data Experiment

📝 Experiment Overview: This experiment implements point cloud data transmission via UDP direct transmission, learning to use the VisionCaptureApi and PX4MavCtrler interfaces, configuring LiDAR sensors, and rendering point cloud images in WSL or a virtual machine.

Experiment 17: Lightweight UAV Model Control Experiment

📝 Experiment Overview: This experiment uses Python to develop a point-mass-based UAV control model, replacing software/hardware-in-the-loop simulation, reducing performance usage, achieving smooth UAV dynamic control, and supporting VS Code debugging and code modification.

Experiment 18: VMware Virtual Machine Ubuntu Environment Configuration

📝 Experiment Overview: Master the basic operations of VMware virtual machines, learn to configure network modes (Bridged and NAT), understand virtual machine settings and login methods, preparing for the use of the RflySim toolchain.

Experiment 19: Livox Mid360 LiDAR Simulation and ROS Visualization

📝 Experiment Overview: Based on the RflySim simulation environment, fully run through the Livox Mid360 LiDAR data link, including sensor simulation, UDP communication configuration, ROS topic forwarding, and RViz point cloud visualization, and achieve Offboard takeoff and position control via PX4MavCtrlV4.

Experiment 20: Timestamp Acquisition Experiment

📝 Experiment Overview: Obtain timestamp data of the drone in RflySim simulation via the Python interface, learning the use of timestamp listening interfaces, and mastering visual sensor configuration and UE control methods.

Experiment 21: RflySim Fisheye Camera Experiment

📝 Experiment Overview: This experiment demonstrates how to use a fisheye camera on the RflySim platform for simulated vision acquisition and flight controller integration, learning to configure visual sensors, acquire images, control the drone, and analyze imaging characteristics.

Experiment 22: Simulated Pod UI Control System

📝 Experiment Overview: Implement a simulated pod UI control system through the RflySim platform, learning pod angle, zoom control, and AI target recognition and tracking functions, mastering the visual sensor interaction mechanism.

Experiment 23: Distributed Vision Control Interface Experiment

📝 Experiment Overview: Summarizes distributed vision interface and control interface experiments, helping users understand cross-host image transmission, remote control, and ROS/MAVROS/MAVSDK integration methods, providing an interface foundation for cross-platform multi-machine collaborative scenarios.

Experiment 24: UDP Direct Transmission Uncompressed Distributed Vision Simulation Experiment

📝 Experiment Overview: This experiment transmits images via UDP direct transmission with PNG uncompressed format, receiving images on a remote Linux or Windows system and sending back aircraft control commands, achieving distributed visual simulation.

Experiment 25: Camera Calibration Experiment

📝 Experiment Overview: Acquire RflySim3D images via the Python interface and update camera parameters in real-time, perform monocular camera calibration, and finally obtain camera intrinsic parameters through MATLAB.

Experiment 26: Ubuntu Virtual Machine Environment Configuration

📝 Experiment Overview: Learn the basic configuration of Ubuntu virtual machines, including the installation of ROS1/ROS2, MAVROS, and common libraries, and understand the Bridged and NAT modes of virtual machine network configuration.

Experiment 27: IMU and Camera Data Acquisition Experiment

📝 Experiment Overview: Acquire IMU and camera data via the Python interface, learning to use VisionCaptureApi for image acquisition, sensor configuration, and aircraft control, with support for real-time camera parameter modification.

Experiment 28: LidarAPIPointCloudDemo

📝 Experiment Overview: Acquire LiDAR point cloud data via the Python interface and display it in real-time, learning key knowledge points such as visual sensor configuration, point cloud display, and aircraft control.

Experiment 29: Python Mavsdk Control Experiment

📝 Experiment Overview: This experiment implements aircraft control via the Python mavsdk interface, using ReqCopterSim to automatically obtain the IP address for establishing distributed co-simulation, demonstrating the MAVLink control process.

Experiment 30: Image Acquisition Without CopterSim Experiment

📝 Experiment Overview: This experiment aims to acquire camera image data from RflySim3D via the Python interface without starting CopterSim, learning core functions such as visual sensor configuration, image acquisition, and real-time camera parameter modification.

Experiment 31: ROS Image Data Subscription Experiment

📝 Experiment Overview: Subscribe to image data from RflySim simulation via ROS, learning to use the Python interface to automatically obtain the IP address and establish connections, achieving image acquisition and OpenCV processing in distributed simulation.

Experiment 32: Pixhawk6xNetSim Visual Box Hardware-in-the-Loop Simulation

📝 Experiment Overview: A visual box hardware-in-the-loop simulation experiment based on the Ethernet interface, configuring the Pixhawk 6X flight controller and NX vision box, and implementing aircraft ring-crossing control via the Python interface and ROS.

Experiment 33: UDP Direct Transmission LiDAR Point Cloud Data Experiment

📝 Experiment Overview: Acquire LiDAR point cloud data via UDP direct transmission mode and display it dynamically in a virtual machine, learning distributed simulation and point cloud data processing.

Experiment 34: Vision AI Interface Experiment

📝 Experiment Overview: Summarizes basic vision AI experiments, covering binocular calibration, camera modeling, 3D position calculation, and dataset generation, providing data and model preparation methods for subsequent vision algorithm experiments.

Experiment 35: Image UDP Direct Transmission JPG Compression Experiment

📝 Experiment Overview: This experiment transmits JPG-compressed images via UDP direct transmission, receiving images on a remote Linux or Windows system and sending back control commands, achieving distributed co-simulation, learning image transmission and remote control.

Experiment 36: Camera Calibration and Field of View Calculation Experiment

📝 Experiment Overview: This experiment uses the RflySim platform to calculate camera focal length, intrinsic parameter matrix, and extrinsic parameter matrix at specified resolutions and fields of view, and verifies imaging errors under different fields of view.

Experiment 37: Modifying sys_id for Visual Box Hardware-in-the-Loop Ring-Crossing Simulation

📝 Experiment Overview: This experiment implements hardware-in-the-loop simulation for the visual box (NX+Pixhawk 6x) by modifying the flight controller's sys_id parameter, controlling the aircraft to fly through a ring. The experiment covers both serial port and Ethernet port connection methods, involving flight controller firmware flashing, parameter configuration, MAVLink communication settings, and Python image processing program execution.

Experiment 38: Depth Camera Image Acquisition Experiment

📝 Experiment Overview: Set depth camera parameters via the Python interface, acquire and display depth image data, learning visual sensor configuration and image acquisition methods.

Experiment 39: DJI Livox LiDAR Point Cloud Image Display Experiment

📝 Experiment Overview: Implement DJI Livox LiDAR scanning via the RflySim platform Python interface, acquiring and displaying point cloud data in real-time, learning key knowledge points such as visual sensor configuration, point cloud display, and aircraft control.

Experiment 40: Python Mavros Control Experiment

📝 Experiment Overview: This experiment demonstrates how to use Python and MAVROS to control the aircraft, including software-in-the-loop simulation, WSL visualization interface operations, offboard mode control, and distributed co-simulation experiments.

Experiment 41: NX and Pixhawk6x Joint Hardware-in-the-Loop Simulation

📝 Experiment Overview: Summarizes the typical configuration entry points for the visual hardware-in-the-loop kit, helping users select Jetson NX, vision box, multi-box, or system ID configuration documents based on their hardware solution, providing preparation entry points for subsequent hardware-in-the-loop ring-crossing, image processing, and control experiments.

Experiment 42: UDP Direct Transmission Client-Server Point Cloud Data Transmission Experiment

📝 Experiment Overview: Acquire LiDAR point cloud data from RflySim3D via UDP direct transmission, and perform point cloud data processing and visualization in a Linux/WSL environment, learning visual sensor configuration and distributed co-simulation experiment methods.

Experiment 43: UavVisionRosCtrl Visual ROS Control Ring-Crossing Experiment

📝 Experiment Overview: Based on RflySim and ROS, acquire visual images via the Python interface, use MAVROS to control the drone for ring-crossing flight, covering simulation configuration, program compilation and execution, and parameter adjustment.

Experiment 44: UDP Direct Transmission JPG Compression Multi-Simulation Experiment

📝 Experiment Overview: This experiment transmits images via UDP direct transmission with JPG compression, configuring multiple simulation windows and cameras, enabling remote Linux or Windows to receive images and send back aircraft control commands, learning distributed visual simulation and co-simulation control.

Experiment 45: Distributed UDP Compressed Image Transmission Experiment (Automatic IP Acquisition)

📝 Experiment Overview: This experiment uses UDP direct transmission of PNG-compressed images to achieve distributed image transmission and flight controller command feedback between Windows and Linux (WSL, virtual machines, boards, etc.), and learns automatic IP acquisition and vision interface configuration.

Experiment 46: Depth Image to Point Cloud Experiment

📝 Experiment Overview: Acquire depth image data via the Python interface and convert it to a point cloud image for real-time display, learning visual sensor configuration, point cloud display, and drone control.

Experiment 47: Aircraft, Object, and Camera Information Acquisition Experiment

📝 Experiment Overview: Acquire information about the aircraft, objects, and cameras via the Python interface, learning to use UE4CtrlAPI and PX4MavCtrl for simulation control and data acquisition.

Experiment 48: Method for Obtaining Precise 3D Positions of Camera, Object, and Target Center

📝 Experiment Overview: Obtain the precise 3D positions of the camera, object, and target center by calling RflySim platform interfaces, learning to use the sendUE4Pos function to generate an aircraft and a ball, and calculating relative positions.

Experiment 49: Python Mavros Control Experiment

📝 Experiment Overview: Use Python and Mavros to implement drone simulation control through the RflySim toolchain, covering software-in-the-loop simulation, WSL visualization interface operations, offboard mode control, and distributed co-simulation experiments.

Experiment 50: Multi-Visual Box Joint Simulation Ring-Crossing Experiment

📝 Experiment Overview: Use two vision boxes to implement joint simulation, controlling the aircraft to fly through a ring, covering hardware configuration, serial port communication, parameter settings, and Python interface usage.

Experiment 51: Point Cloud Visualization Experiment

📝 Experiment Overview: Implement point cloud data acquisition and visualization through RflySim platform interfaces, supporting shared memory and UDP transmission modes, and learn to use relevant Python interfaces for distributed co-simulation.

Experiment 52: Data UDP Direct Transmission PNG Compression Experiment

📝 Experiment Overview: Learn to transmit images using UDP direct transmission with PNG compression, receive images on a remote Linux or Windows system and send back aircraft control commands, mastering SendProtocol parameter configuration and vision interface usage.

Experiment 53: IMU Data Acquisition Experiment

📝 Experiment Overview: Acquire IMU data from CopterSim simulation via the Python vision interface, learning to use VisionCaptureApi to send IMU data requests and read data, mastering simulation environment configuration and debugging methods.

Experiment 54: Automatic Generation of YOLO Dataset Experiment
  • 📦 Version Requirement: Free Edition
  • **📁

8.5.2 Basic Usage Experiments

Stored in the 8.RflySimVision\1.BasicExps folder, these experiments provide a complete set of supplementary teaching materials for beginners.

Experiment 1: Point-Mass Model Ring-Crossing Experiment

📝 Experiment Overview:
Acquire RflySim 3D images via the Python interface and update camera parameters in real-time to implement a point-mass model-based drone visual ring-crossing experiment. Learn the use of the vision interface and camera parameter configuration.

Experiment 2: Drone Visual Tracking of a Ball

📝 Experiment Overview:
Acquire images via the RflySim platform, control the drone to visually track and stop in front of a red ball. Learn the vision interface, camera configuration, aircraft control, and image processing algorithms.

Experiment 3: Drone Vision-Based Ring-Crossing Control Experiment

📝 Experiment Overview:
This experiment utilizes the RflySim platform's vision interface, employing OpenCV object recognition and visual servoing control to enable the drone to autonomously fly through colored rings and square frames. It covers single-drone, dual-drone, and three-drone distributed cooperative ring-crossing tasks.

Experiment 4: Drone Visual Following of a Circular Target

📝 Experiment Overview:
Demonstrate computer vision-based autonomous following technology by controlling a drone to track a moving circular target. Learn target detection, image processing, visual servoing control, and PID control applications.

Experiment 5: Binocular Vision Face Recognition Experiment

📝 Experiment Overview:
Configure binocular grayscale cameras via the RflySim platform, acquire images, and implement face recognition. Learn the use of the vision interface, camera parameter configuration, and aircraft control.

Experiment 6: Screen Capture API Ball Collision Experiment

📝 Experiment Overview:
This experiment uses the RflySim platform's screen capture API to acquire RflySim 3D window images, performs visual processing to control the drone to collide with a ball. Learn the screen capture image acquisition method and the visual control process.

Experiment 7: Basic Vision Control Experiment

📝 Experiment Overview:
A collection of basic vision control experiments. Through routines such as ring-crossing, ball collision, target following, gimbal control, and multi-drone cooperation, users can understand the visual control closed loop and provide reproducible basic control cases for complex algorithm experiments.

Experiment 8: RflySim Platform Basic Vision Competition Interface Experiment

📝 Experiment Overview:
Validate the end-to-end data flow of perception pipelines including cameras, depth cameras, and LiDAR. Practice the closed loop from perception to motion control, become familiar with ROS topics, coordinate system mapping, and the MAVROS control interface, and set up a sample environment for algorithm development in visual SLAM, navigation, and obstacle avoidance.

Experiment 9: Drone Vision-Based Ring-Crossing Control

📝 Experiment Overview:
Acquire images via the screen capture API to complete drone vision-based ring-crossing control. Validate the complete closed loop from screen capture, target recognition, to flight control output, and become familiar with the experimental procedures under both software-in-the-loop and hardware-in-the-loop startup modes.

Experiment 10: RflySim Platform Basic Functionality Demo

📝 Experiment Overview:
This experiment configures multiple sensors (RGB camera, depth camera, LiDAR, etc.) in the RflySim simulation environment, implements real-time cross-platform (Windows and Ubuntu) data transmission, and uses ROS for sensor data subscription, processing, and visualization to master multi-sensor fusion and cross-platform communication technologies.

Experiment 11: Drone Recognition and Path Planning Competition Experiment

📝 Experiment Overview:
This experiment controls the drone to take off, recognize environmental objects in real-time and plan a path to fly through a square frame, identify a ground vehicle carrying an ArUco marker and output its ID, and finally recognize the landing position and land accurately to complete the competition task.

Experiment 12: LLM-Based Drone Swarm Control Experiment

📝 Experiment Overview:
Master the method of controlling drones using Large Language Models (LLMs). Learn to implement drone takeoff, frame crossing, QR code recognition, landing, and other operations through natural language commands. Understand the application principles and practical methods of large models in drone intelligent control systems.

Experiment 13: LLM-Based Drone Racing SLAM Experiment

📝 Experiment Overview:
This experiment uses LiDAR SLAM (Mid-360 solid-state LiDAR sensor) combined with an IMU for autonomous drone localization without GNSS, and integrates an LLM for environmental understanding and decision-making, achieving drone racing flight and real-time point cloud mapping.

Experiment 14: Air-Ground Cooperative Track Simulation Operation

📝 Experiment Overview:
Based on the RflySim simulation platform, this experiment guides users through deploying and running air-ground cooperative (drone and ground vehicle) simulation routines, covering core processes such as environment configuration, sensor data acquisition, SLAM localization, and motion control.

Experiment 15: Drone Following a Circular Board

📝 Experiment Overview:
Acquire RflySim 3D images via the vision interface, use keyboard input to control the movement of a circular board, and implement drone visual following control.

Experiment 16: Shared Memory Gimbal Vision Keyboard Control Simulation Experiment

📝 Experiment Overview:
Control the drone gimbal vision (pitch, yaw, roll, focal length) via keyboard. Learn shared memory image transmission, sensor configuration, and Python keyboard control methods.

Experiment 17: Three-UAV Distributed Control Experiment

📝 Experiment Overview:
Use three Python scripts to control three drones to perform ring-crossing flights sequentially. Covers key knowledge points including software/hardware-in-the-loop simulation, vision sensor configuration, drone control, and image processing.

Experiment 18: Dual-UAV Distributed Control Experiment

📝 Experiment Overview:
Use two Python scripts to control two drones to fly through rings sequentially. Learn screen capture image acquisition, image processing, and distributed control techniques.

8.5.3 Advanced Development Experiments

Stored in the 8.RflySimVision\2.AdvExps folder, these experiments further familiarize users with certain low-level firmware ecosystem configurations.

Experiment 1: UDP Direct Transmission Gimbal Vision Keyboard Control Simulation Virtual Machine Experiment

📝 Experiment Overview: Run the program in a virtual machine, receive image data via UDP direct transmission, and use the keyboard to control the gimbal's pitch, yaw, roll angles, and field of view. Learn visual sensor configuration and simulation environment setup.

Experiment 2: UDP Direct Transmission Gimbal Vision Keyboard Control Simulation Experiment

📝 Experiment Overview: Use the platform interface and keyboard to control the gimbal vision's pitch, yaw, roll angles, and focal length. Learn sensor configuration and UDP direct image transmission methods.

Experiment 3: Gimbal Vision Keyboard Control ROS Simulation Experiment

📝 Experiment Overview: Implement keyboard control of gimbal camera angles and focal length via ROS, with UDP communication between the platform and WSL. Learn distributed simulation and vision control.

Experiment 4: RflySim Platform Vision SLAM Experiment

📝 Experiment Overview: Based on the RflySim toolchain, this experiment achieves UAV visual SLAM control and automatic obstacle avoidance through collaboration between a Windows client and an Ubuntu virtual machine/board server. Learn visual sensor configuration, image transmission, and distributed simulation interface usage.

Experiment 5: LiDAR SLAM Experiment

📝 Experiment Overview: This experiment uses the RflySim platform and LiDAR SLAM algorithms to achieve automatic obstacle avoidance movement for UAVs in a simulated environment. Learn the vision interface, ReqCopterSim interface, and distributed co-simulation configuration.

Experiment 6: YOLO-Based Object Detection Experiment

📝 Experiment Overview: A collection of YOLO-based object detection experiments, helping users understand the complete pipeline from image acquisition, object detection, to flight control or sensor output. Provides an algorithmic foundation for experiments such as visual servoing and target following.

Experiment 7: Single Object Tracking Experiment

📝 Experiment Overview: Acquire images via the RflySim platform interface, and use object tracking algorithms to control the UAV for real-time tracking of target objects.

Experiment 8: Target Following Visual Servoing Control Experiment

📝 Experiment Overview: Use the target results directly output by the platform as input, and employ visual servoing algorithms to control the UAV to track highly maneuverable targets. Validate the visual servoing control pipeline and provide reproducible examples for debugging object detection, tracking, and visual servoing algorithms.

Experiment 9: Astar Path Planning Experiment

📝 Experiment Overview: A collection of Astar path planning experiments, helping users understand path planning implementation based on grid search and ROS point cloud input. Provides an algorithmic foundation for advanced vision experiments in path planning, obstacle avoidance, and navigation.

Experiment 10: KCF Ring-Through and Companion Flight Experiment

📝 Experiment Overview: This experiment evaluates the integration of the KCF object recognition and tracking algorithm, the EGO-Planner path planning algorithm, and the RflySim toolchain by having the UAV perform two tasks: flying through ring gates and accompanying a ground vehicle.

Experiment 11: EGO-Swarm Visual Swarm Perception and Planning Experiment

📝 Experiment Overview: Evaluate the integration of the EGO-Swarm path planning algorithm with the RflySim toolchain through a task where three UAVs traverse a forest. Learn multi-UAV distributed collaborative perception and planning.

Experiment 12: UAV LiDAR Real-Time 3D Point Cloud Mapping

📝 Experiment Overview: Utilize LiDAR and odometry data from the RflySim simulation platform to perform point cloud coordinate transformation and stitching in a ROS environment. Construct a 3D solid map using the TF transform tree and homogeneous transformation matrices.

Experiment 13: Dual-UAV TF Tree Construction Experiment

📝 Experiment Overview: This experiment uses the RflySim simulation platform to learn how to construct a TF tree for a dual-UAV system in a ROS environment. Master coordinate transformation, spatial relationship modeling, and namespace isolation mechanisms.

Experiment 14: UAV Path Planning Experiment Based on ESDF and Voronoi Diagrams

📝 Experiment Overview: This experiment processes a 2D grid map to construct an ESDF field and Voronoi diagram, performs path planning, and maps the trajectory onto the RflySim platform to control the UAV for closed-loop obstacle avoidance movement.

Experiment 15: UAV Autonomous Following Based on YOLO Visual Detection

📝 Experiment Overview: This experiment completes a full closed loop in the RflySim visual simulation environment: data collection, YOLO object detection model training, and UAV vision-based following control. Learners will master visual servoing control strategies.

Experiment 16: YOLO-Based Balloon Shooting Experiment

📝 Experiment Overview: Acquire images via the RflySim platform interface, use the YOLO algorithm to detect balloons, and control the UAV to collide with the balloon. Learn visual sensor configuration, image transmission, and UAV control.

Experiment 17: Simple YOLO Sensor Multi-UAV Recognition

📝 Experiment Overview: Identify UAVs using a simple YOLO sensor, become familiar with its configuration method, and apply it to large-scale simulation training.

Experiment 18: AStar Algorithm Path Planning Experiment

📝 Experiment Overview: Use the A* algorithm for path planning, modifying the traditional 4-neighborhood search to an 8-neighborhood search. Involves image processing, coordinate transformation, and OpenCV applications. Achieve UAV path planning simulation through the RflySim toolchain.

Experiment 19: LiDAR SLAM Vision Image Acquisition and IMU Data Collection Experiment

📝 Experiment Overview: Learn to run the RflySim LiDAR SLAM vision image acquisition demonstration, modify configurations to adjust UE frame rate and image acquisition frequency, analyze image and IMU timestamps, and evaluate image acquisition latency for UAV control.

Experiment 20: RflySim Platform SLAM Image Acquisition Latency Analysis

📝 Experiment Overview: Conduct SLAM vision image acquisition experiments on the RflySim platform, analyze image and IMU timestamp latency, and adjust UE4 frame rate and image acquisition frequency to optimize latency.

Experiment 21: Visual SLAM Image Acquisition Latency Analysis

📝 Experiment Overview: Run the visual SLAM demonstration to analyze the latency between UE4 image acquisition frequency and IMU timestamps. Learn to configure MaxFPS and DataCheckFreq to optimize image acquisition performance.

Experiment 22: Simple YOLO Sensor Usage Experiment

📝 Experiment Overview: Identify UAVs using a simple YOLO sensor, become familiar with its configuration method, and understand how the simple sensor and RGB sensor jointly output object recognition results. Provides a foundational sensor solution for large-scale simulation training and multi-object recognition experiments.

Experiment 23: Astar Algorithm Path Planning (ROS) Experiment

📝 Experiment Overview: Based on the ROS platform, use the A* algorithm for global path planning, and communicate with the flight controller via the MAVROS interface to enable the UAV to fly along the planned path. Covers steps such as code compilation, program startup, and TF coordinate transformation configuration.

Experiment 24: YOLOv5 Object Detection REST API Service

📝 Experiment Overview: Build a REST API using the Flask framework to expose the YOLOv5 object detection model, enabling other services to call the PyTorch Hub YOLOv5s model for image object detection inference via HTTP requests.

Experiment 25: YOLOv5 Integrated with W&B Visualization Training Tool

📝 Experiment Overview: Introduces how to integrate the Weights & Biases (W&B) machine learning experiment tracking tool into YOLOv5, enabling training process visualization, metric monitoring, dataset version management, and experiment comparison functionality.

8.5.4 Advanced Development Experiments

Stored in the 8.RflySimVision\3.CustExps folder, these are custom development experiments designed for advanced users.

Experiment 1: VINS-Fusion Visual SLAM Mapping

📝 Experiment Overview:
Run the VINS-Fusion algorithm in a Linux environment, utilizing RflySim simulation data from the Windows platform to achieve visual SLAM mapping for UAVs. Learn the principles of the VINS-Fusion algorithm and its simulation integration.

Experiment 2: Integration and Validation of ORB-SLAM3 with RflySim Simulation Data

📝 Experiment Overview:
Learn to integrate ORB-SLAM3 as a ROS node with RflySim data streams (images and IMU), build and run an ORB-SLAM3 ROS node locally, and collect trajectory and pose data for publishing.

Experiment 3: UAV Control Using Behavior Trees

📝 Experiment Overview:
This experiment uses behavior trees to implement UAV takeoff, point-to-point flight, and landing functions, and demonstrates how to customize behavior tree nodes. Learn the basic concepts of behavior trees, MAVROS communication mechanisms, and custom node development.

Experiment 4: Cooperative Reconnaissance and Strike by Dual UAVs

📝 Experiment Overview:
Simulate a cooperative dual-UAV combat scenario: UAV #2 performs reconnaissance to locate a red balloon target, transmits the target information to UAV #1, and UAV #1 takes off and strikes the target. This validates the system's capabilities in information sharing, target recognition, and precise strike during cooperative operations.

Experiment 5: Behavior Tree Combined with Large Model Visual Control

📝 Experiment Overview:
Deploy a quantized Qwen model locally via Ollama, parse natural language commands to generate behavior trees, and combine ROS with PX4 SITL to achieve a closed-loop visual control system for UAVs. Learn the mapping from semantics to actions.

Experiment 6: Audio-Visual Sensor Fusion Experiment

📝 Experiment Overview:
Master audio-visual sensor fusion recognition techniques. Through ITD-based sound source localization and YOLOv5 visual detection, achieve UAV tracking control of targets.

Experiment 7: Air-Ground Cooperative Software-in-the-Loop Simulation Operation

📝 Experiment Overview:
Based on the Air-Ground Cooperative Track of the 28th China Robot Competition, this experiment provides complete code and operation guides for software-in-the-loop simulation (SITL) of both UAVs and ground vehicles. It covers environment setup, compilation, launching the simulation platform, and running programs to accomplish tasks such as the UAV flying through a square frame, popping balloons, and the ground vehicle passing through a gate.