VisionCaptureApi Interface Documentation

Introduction

Overview: This document defines the data structures and API classes required by the visual perception module of the RflySim UAV simulation platform, providing foundational interactive support for Python-side requests to the UE4 simulation environment for camera images and various sensor data.

In RflySim simulation, visual tasks are central to the development of intelligent algorithms such as UAV autonomous navigation and target detection, requiring parameter exchange and data transmission between the Python and UE4 sides. This module defines structures for sending camera parameter settings and image requests to UE4, as well as encapsulation structures for various perception data—including IMU data and distance sensor data—from CopterSim. It also provides the core VisionCaptureApi class, enabling Python-side acquisition of simulated images from the UE4 simulation environment. It is suitable for developing vision-based UAV simulation algorithms, offering a unified interface specification for visual perception data interaction in simulation environments, supporting monocular and multi-camera simulated data acquisition. Combined with IMU and distance sensor information, it enables multi-sensor fusion simulation tasks.

Quick Start

Minimal working example; copy and modify minimal configuration to run.

from RflySimSDK.vision.VisionCaptureApi import VisionCaptureApi, VisionSensorReqNew

# 1. Initialize the visual capture API; connects to local RflySim simulation environment by default
vision_api = VisionCaptureApi(ip="127.0.0.1")

# 2. Create a new visual sensor request object; requests RGB image and pose data by default
sensor_req = VisionSensorReqNew()

# 3. Add the visual sensor request to the capture list
try:
    vision_api.addVisSensor(sensor_req)
except Exception as e:
    print(f"Failed to add visual sensor: {e}")

# 4. Euler angle to quaternion conversion example: roll=0°, pitch=0°, yaw=90°, converted to quaternion
roll = 0
pitch = 0
yaw = 90
quat = vision_api.euler2quat(roll, pitch, yaw, copter="iris", coordinate_frame="global")
print(f"Converted quaternion: {quat}")

Environment and Dependencies

• Python Environment: >= 3.8.10
• Dependencies: Open3DShow, copy, ctrl.IpManager, cv2, email, json, math, mmap, numpy, os, platform, psutil, re, socket, struct, subprocess, sys, threading, time, warnings
• Prerequisites: Before calling this interface, ensure that the RflySimSDK vision module has been correctly imported.

Core Interface Description

The module VisionCaptureApi.py includes configuration variables, helper functions, and the core business class.

Global Constants and Enumerations

This section lists all globally accessible constants and enumeration definitions directly referenceable within the module.

Standalone Constants

None

---

Global/Standalone Functions

None

---

Queue Class

Provides a basic queue data structure for storing and managing elements in a first-in-first-out (FIFO) manner.

__init__()

Function Description: Initializes an empty queue instance
Parameters (Args):
None
Returns:

• Queue instance object

Exceptions (Raises):
None

---

enqueue(item)

Function Description: Adds an element to the tail of the queue
Parameters (Args):

Parameter Name	Type	Required	Default	Description
item	Any type	Yes	-	Element to be added to the queue

Returns:

• None

Exceptions (Raises):
None

---

dequeue()

Function Description: Removes and returns the element at the head of the queue
Parameters (Args):
None
Returns:

• Element at the head of the queue

Exceptions (Raises):
None

---

is_empty()

Function Description: Determines whether the current queue is empty
Parameters (Args):
None
Returns:

• bool: Returns True if the queue is empty, otherwise returns False

Exceptions (Raises):
None

---

size()

Function Description: Retrieves the current number of elements stored in the queue
Parameters (Args):
None
Returns:

• int: Number of elements in the queue

Exceptions (Raises):
None

Example:

from RflySimSDK.vision import Queue

# Create a queue and add elements
q = Queue()
q.enqueue("first frame")
q.enqueue("second frame")

# Check queue size
print(q.size())  # Outputs: 2

# Dequeue an element
item = q.dequeue()
print(item)  # Outputs: first frame

# Check if queue is empty
print(q.is_empty())  # Outputs: False

RflyTimeStmp Class

A utility class for managing timestamp data of drone vision data, capable of storing and updating timestamp values, used in conjunction with the vision capture module.

__init__(iv)

Function Description: Initializes an RflyTimeStmp instance and sets the initial timestamp value.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
iv	Any numeric type	Yes	-	Initial timestamp value

Return Value (Returns):

• RflyTimeStmp instance object

Exceptions (Raises): - None

---

Update(iv)

Function Description: Updates the stored timestamp value to the new input value.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
iv	Any numeric type	Yes	-	New timestamp value to be updated

Return Value (Returns): - None
Exceptions (Raises): - None

Example:

from RflySimSDK.vision import RflyTimeStmp
# Initialize timestamp to 0
ts = RflyTimeStmp(0)
# Update timestamp to current simulation time 12.34
ts.Update(12.34)

---

VisionSensorReq Class

A C++ structure corresponding to a Python class used to request and configure camera parameters in the UE4 simulation environment; applicable for configuring vision sensor parameters in RflySim drone vision simulation.

__init__()

Function Description: Initializes a VisionSensorReq instance; all parameter members are created according to the default C++ structure definition.
Parameters (Args): - None
Return Value (Returns):

• VisionSensorReq instance object

Exceptions (Raises): - None

Example:

from RflySimSDK.vision import VisionSensorReq
# Create a vision sensor parameter request object
sensor_req = VisionSensorReq()
# Configure the target aircraft ID
sensor_req.TargetCopter = 1
# Set the camera mounting position
sensor_req.SensorPosXYZ = [0, 0, 0.1]

---

VisionSensorReqNew Class

A C++-style structure used to send requests and configure vision sensor parameters to the UE4 simulation environment; used in RflySim drone vision simulation to define vision sensor configuration information.

__init__()

Function Description: Initializes a VisionSensorReqNew instance object.
Parameters (Args): - None
Return Value (Returns):

• VisionSensorReqNew instance object

Exceptions (Raises): - None

---

imuDataCopter Class

A data structure used to receive IMU sensor data from CopterSim, storing checksum, message sequence number, timestamp, accelerometer data, and gyroscope data.

__init__(imu_name, node=None)

Function Description: Initializes an imuDataCopter instance, setting the IMU data ROS topic name and ROS node.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
imu_name	str	No	"/rflysim/imu"	ROS topic name for IMU data
node	Any	No	None	ROS node object used to publish IMU data

Return Value (Returns):

• imuDataCopter instance object

Exceptions (Raises):

• None

---

AlignTime(img_time)

Function Description: Aligns the IMU data timestamp with the image timestamp to match IMU data at the corresponding moment.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
img_time	double	Yes	-	Image timestamp to be aligned

Return Value (Returns):

• Aligned and matched IMU data

Exceptions (Raises):

• None

---

Imu2ros(node=None)

Function Description: Converts the current IMU data into ROS format and publishes it.
**Parameters (

Parameter Name	Type	Required	Default Value	Description
r	Any	Yes	None	Roll angle Euler component
p	Any	Yes	None	Pitch angle Euler component
y	Any	Yes	None	Yaw angle Euler component
copter	str	Yes	None	Vehicle type identifier
coordinate_frame	Any	No	"global"	Coordinate frame type; defaults to global coordinate frame

Return Value (Returns): The converted quaternion
Exceptions (Raises): None

---

addVisSensor(vsr=VisionSensorReq())

Function Description: Adds a new legacy VisionSensorReq visual sensor request structure to the visual sensor list
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vsr	Any	No	VisionSensorReq()	Legacy visual sensor request structure to be added

Return Value (Returns): None
Exceptions (Raises): Exception — thrown if addition fails

---

addVisSensor(vsr=VisionSensorReqNew())

Function Description: Adds a new updated VisionSensorReqNew visual sensor request structure to the visual sensor list
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vsr	Any	No	VisionSensorReqNew()	Updated visual sensor request structure to be added

Return Value (Returns): None
Exceptions (Raises): Exception — thrown if addition fails

---

sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")

Function Description: Sends a SensorReqCopterSim-type UDP request message to CopterSim to request sensor data; the target CopterSim instance is specified via copterID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
srcs	Any	No	SensorReqCopterSim()	Sensor request structure
copterID	Any	No	1	ID of the target vehicle; specifies the index of the CopterSim instance to request data from
IP	Any	No	"127.0.0.1"	IP address of the host where the target CopterSim is running

Return Value (Returns): None
Exceptions (Raises): None

---

sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)

Function Description: Sends an IMU data request command to CopterSim; this function initializes a thread to listen for incoming IMU data.
The actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	Any	No	1	ID of the target vehicle
IP	Any	No	"127.0.0.1"	IP address of the target host to which the request is sent
freq	Any	No	200	Requested IMU data transmission frequency

Return Value (Returns): None
Exceptions (Raises): None

---

sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)

Function Description: Sends an IMU data request command to CopterSim. The actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	Any	No	1	ID of the target vehicle
IP	Any	No	"127.0.0.1"	IP address of the target host to which the request is sent
freq	Any	No	200	Requested IMU data transmission frequency

Return Value (Returns): None
Exceptions (Raises): None

---

sendImuReqServe(copterID=1)

Function Description: Sends an IMU data request command to CopterSim; this function initializes a thread to listen for and receive IMU data, where the actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	Any	No	1	ID of the target vehicle

Return Value (Returns): None
Exceptions (Raises): None

---

getIMUDataLoop(copterID)

Function Description: Cyclically retrieves IMU data for a specified vehicle, serving as the worker function for the IMU listening thread.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	Any	Yes	None	ID of the vehicle whose data is to be retrieved

Return Value (Returns): None
Exceptions (Raises): None

---

get_all_ip()

Function Description: Retrieves all available network interface IP addresses on the local machine.
Parameters (Args): None
Return Value (Returns): A list of all IP addresses on the local machine
Exceptions (Raises): None

---

isIpLocal(IP)

Function Description: Determines whether a given IP address is a local machine IP address.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
IP	Any	Yes	None	The IP address to be checked

Return Value (Returns): bool — returns True if the IP is local, otherwise False
Exceptions (Raises): None

---

StartTimeStmplisten()

Function Description: Starts a listening thread to monitor port 20005 for Rfly time stamps corresponding to various vehicle IDs.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

---

getTimeStmp(CopterID=1)

Function Description: Retrieves the latest Rfly time stamp corresponding to a specified vehicle ID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
CopterID	Any	No	1	ID of the target vehicle

Return Value (Returns): The latest Rfly time stamp for the specified vehicle
Exceptions (Raises): None

---

endTimeStmplisten()

Function Description: Terminates the time stamp listening thread, stopping the reception of time stamp data.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

---

TimeStmploop()

Function Description: The worker function of the time stamp listening thread, continuously receiving and processing time stamp data sent by CopterSim.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

---

sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")

Function Description: Sends a visual sensor image update request to UE4.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vs	Any	No	VisionSensorReq()	Visual sensor request structure
windID	Any	No	0	Window ID identifier
IP	Any	No	""	IP address of the host running the target UE4; if empty, automatically determined

Return Value (Returns): None
Exceptions (Raises): Exception — thrown when the request fails to send

---

sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")

Function Description: Updates the visual sensor configuration for a specified RflySim3D window.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vs	-	No	VisionSensorReqNew()	Visual sensor request configuration object
windID	-	No	0	RflySim3D window index
IP	-	No	""	Target IP address; uses default if empty

Return Value (Returns): No return value
Exceptions (Raises): None

---

sendImgUDPNew(idx)

Function Description: Sends the image of the specified visual sensor index via the new UDP protocol.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
idx	-	Yes	None	Visual sensor index

Return Value (Returns): No return value
Exceptions (Raises): None

---

sendImgBuffer(idx, data)

Function Description: Sends the image buffer data of the specified visual sensor index.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
idx	-	Yes	None	Visual sensor index
data	-	Yes	None	Image data buffer to be sent

Return Value (Returns): No return value
Exceptions (Raises): None

---

jsonLoad(ChangeMode=-1, jsonPath="")

Function Description: Loads a configuration JSON file and creates a camera list for image acquisition. If ChangeMode >= 0, the first item SendProtocol[0] is set to ChangeMode to modify the image transmission mode.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
ChangeMode	-	No	-1	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration
jsonPath	-	No	""	Path to the configuration JSON file; uses default path if empty

Return Value (Returns): No return value
Exceptions (Raises): None

---

dictLoad(jsData, ChangeMode=-1)

Function Description: Wrapper for the jsonLoad method, enabling direct reading of sensor data stored in memory as a dictionary. The data structure matches that of the JSON file used in jsonLoad.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
jsData	-	Yes	None	Dictionary of sensor configuration data stored in memory
ChangeMode	-	No	-1	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration

Return Value (Returns): No return value
Exceptions (Raises): None

---

stopRun()

Function Description: Stops the image acquisition runtime loop and terminates all image-reception threads.
Parameters (Args): No parameters
Return Value (Returns): No return value
Exceptions (Raises): None

---

SensorPreview(seq_id=-1)

Function Description: Previews images captured by the visual sensor with the specified sequence ID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
seq_id	-	No	-1	Sequence ID of the visual sensor to preview; -1 indicates previewing all sensors

Return Value (Returns): No return value
Exceptions (Raises): None

Advanced Usage Example

Demonstrates complex composite scenarios (e.g., multi-type collaboration, asynchronous control, batch operations)

from RflySimSDK.vision import VisionCaptureApi, Queue, imuDataCopter
import threading

# Initialize a queue to buffer sensor data, enabling asynchronous data reception
imu_data_queue = Queue()
vision_capture = VisionCaptureApi()

# Batch add 5 heterogeneous visual sensors to the simulation environment
for sensor_id in range(5):
    # Customize installation poses for different sensors, automatically converting to quaternions
    roll, pitch, yaw = 0.1 * sensor_id, 0.05, -0.02 * sensor_id
    quat = vision_capture.euler2quat(roll, pitch, yaw)
    vision_capture.addVisSensor(sensor_id=sensor_id, position=[0.1 * sensor_id, 0, 0.2], quaternion=quat)

# Start an asynchronous IMU data acquisition thread; after time alignment, store data in a queue
def get_imu_async():
    imu_handler = imuDataCopter()
    while True:
        raw_imu = vision_capture.getIMUDataLoop()
        if raw_imu is not None:
            # Align IMU timestamps with visual frame timestamps
            aligned_imu = imu_handler.AlignTime(raw_imu)
            if not imu_data_queue.is_empty() and imu_data_queue.size() > 10:
                imu_data_queue.dequeue()
            imu_data_queue.enqueue(aligned_imu)

# Start asynchronous thread for data processing; main thread synchronously requests visual frames for SLAM computation
threading.Thread(target=get_imu_async, daemon=True).start()
vision_capture.sendReqToCopterSim()

Notes and Pitfall Avoidance Guide

• Duplicate Visual Sensor Addition Issue: When calling addVisSensor multiple times, ensure the input sensor_id is globally unique; duplicate IDs will cause the simulation side to overwrite sensors, resulting in erroneous visual data retrieval.
• Asynchronous Data Buffer Overflow: When using a custom queue to cache IMU or visual data, periodically clean up old data; prolonged operation will cause the queue to accumulate and consume excessive memory. This can be mitigated by checking the queue size and popping the oldest data from the front once it exceeds a threshold.
• Cross-Host IP Connection Check: Before establishing a connection, call isIpLocal to detect IP properties. If the simulation side runs on a non-local host, ensure the IP port permissions are open; otherwise, the default port may be blocked by the remote firewall, causing request timeouts.
• Prerequisites for Time Alignment: Before calling imuDataCopter.AlignTime, time stamp initialization must be completed first. Failure to call RflyTimeStmp.Update to update the reference time stamp will result in alignment deviations, causing SLAM pose jumps.

Changelog

• 2026-03-03: feat: SDK adds IP handling mechanism, compatible with cloud deployment from local version
• 2026-02-02: feat: Add odom publishing for depth-to-point-cloud conversion
• 2026-02-02: fix: Optimize protocol
• 2026-02-01: fix: Comment out path printing
• 2026-01-31: fix: Update depth-to-point-cloud protocol
• 2026-01-31: fix: Update communication protocol for depth-to-point-cloud
• 2026-01-23: Align timestamps for /Depth_Cloud topic
• 2026-01-22: Merge branch 'master' of http://rflysim.synology.me:8418/FeisiLab/RflySimSDK
• 2026-01-22: Change IMU coordinate frame orientation for ROS output to ensure consistent FLU frame output; previously, conversion was FRD → FLU, but the platform’s IMU angular velocity conforms to FRD, while linear acceleration does not conform to FRD but rather BLD
• 2025-12-12: fix: Optimize image timestamp synchronization mechanism, resolve image timestamp desynchronization bug on Windows
• 2025-12-08: Change platform odom data translation in local frame under mode 0 from FRD → FLU
• 2025-12-07: fix: Add relative position transmission data for LiDAR
• 2025-12-04: fix: Fix point cloud sensor issue
• 2025-09-30: Resolve fisheye camera bug under UDP mode
• 2025-09-29: ROS2 depth-to-point-cloud seq variable has been removed; interface needs adaptation