VisionCaptureApi Interface Documentation¶

Introduction¶

Overview: This document defines the data structures and API classes required by the visual perception module of the RflySim UAV simulation platform, providing foundational interactive support for Python-side requests to the UE4 simulation environment for camera images and various sensor data.

In RflySim simulation, visual tasks are central to the development of intelligent algorithms such as UAV autonomous navigation and target detection, requiring parameter exchange and data transmission between the Python and UE4 sides. This module defines structures for sending camera parameter settings and image requests to UE4, as well as encapsulation structures for various perception data—including IMU data and distance sensor data—from CopterSim. It also provides the core VisionCaptureApi class, enabling Python-side acquisition of simulated images from the UE4 simulation environment. It is suitable for developing vision-based UAV simulation algorithms, offering a unified interface specification for visual perception data interaction in simulation environments, supporting monocular and multi-camera simulated data acquisition. Combined with IMU and distance sensor information, it enables multi-sensor fusion simulation tasks.

Quick Start¶

The following example loads sensor configuration from Config.json, sends an image request to RflySim3D, and displays the received images from each channel.

Reference example: [RflySim installation path]\RflySimAPIs\8.RflySimVision\0.ApiExps\1-UsageAPI\0.ConfigJsonAPI

import sys

import cv2
import VisionCaptureApi


vis = VisionCaptureApi.VisionCaptureApi()
vis.jsonLoad()

for sensor in vis.VisSensor:
    print("SeqID:", sensor.SeqID)
    print("SendProtocol:", sensor.SendProtocol)

if not vis.sendReqToUE4():
    sys.exit(0)
vis.startImgCap()

while True:
    for index in range(len(vis.hasData)):
        if vis.hasData[index]:
            cv2.imshow("Img" + str(index), vis.Img[index])
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

Environment and Dependencies¶

Python Environment: >= 3.8.10
Dependencies: Open3DShow, copy, ctrl.IpManager, cv2, email, json, math, mmap, numpy, os, platform, psutil, re, socket, struct, subprocess, sys, threading, time, warnings
Prerequisites: Before calling this interface, ensure that the RflySimSDK vision module has been correctly imported.

Core Interface Description¶

The module VisionCaptureApi.py includes configuration variables, helper functions, and the core business class.

Global Constants and Enumerations¶

This section lists all globally accessible constants and enumeration definitions directly referenceable within the module.

Standalone Constants¶

None

Global/Standalone Functions¶

None

`Queue` Class¶

Provides a basic queue data structure for storing and managing elements in a first-in-first-out (FIFO) manner.

`init()`¶

Function Description: Initializes an empty queue instance
Parameters (Args):
None
Returns:

Queue instance object

Exceptions (Raises):
None

`enqueue(item)`¶

Function Description: Adds an element to the tail of the queue
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`item`	`Any type`	Yes	-	Element to be added to the queue

Returns:

None

Exceptions (Raises):
None

`dequeue()`¶

Function Description: Removes and returns the element at the head of the queue
Parameters (Args):
None
Returns:

Element at the head of the queue

Exceptions (Raises):
None

`is_empty()`¶

Function Description: Determines whether the current queue is empty
Parameters (Args):
None
Returns:

bool: Returns True if the queue is empty, otherwise returns False

Exceptions (Raises):
None

`size()`¶

Function Description: Retrieves the current number of elements stored in the queue
Parameters (Args):
None
Returns:

int: Number of elements in the queue

Exceptions (Raises):
None

Example:

from RflySimSDK.vision import Queue

# Create a queue and add elements
q = Queue()
q.enqueue("first frame")
q.enqueue("second frame")

# Check queue size
print(q.size())  # Outputs: 2

# Dequeue an element
item = q.dequeue()
print(item)  # Outputs: first frame

# Check if queue is empty
print(q.is_empty())  # Outputs: False

¶

`RflyTimeStmp` Class¶

A utility class for managing timestamp data of drone vision data, capable of storing and updating timestamp values, used in conjunction with the vision capture module.

`init(iv)`¶

Function Description: Initializes an RflyTimeStmp instance and sets the initial timestamp value.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`iv`	Any numeric type	Yes	-	Initial timestamp value

Return Value (Returns):

RflyTimeStmp instance object

Exceptions (Raises): - None

`Update(iv)`¶

Function Description: Updates the stored timestamp value to the new input value.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`iv`	Any numeric type	Yes	-	New timestamp value to be updated

Return Value (Returns): - None
Exceptions (Raises): - None

Example:

from RflySimSDK.vision import RflyTimeStmp
# Initialize timestamp to 0
ts = RflyTimeStmp(0)
# Update timestamp to current simulation time 12.34
ts.Update(12.34)

`VisionSensorReq` Class¶

A C++ structure corresponding to a Python class used to request and configure camera parameters in the UE4 simulation environment; applicable for configuring vision sensor parameters in RflySim drone vision simulation.

`init()`¶

Function Description: Initializes a VisionSensorReq instance; all parameter members are created according to the default C++ structure definition.
Parameters (Args): - None
Return Value (Returns):

VisionSensorReq instance object

Exceptions (Raises): - None

Example:

from RflySimSDK.vision import VisionSensorReq
# Create a vision sensor parameter request object
sensor_req = VisionSensorReq()
# Configure the target aircraft ID
sensor_req.TargetCopter = 1
# Set the camera mounting position
sensor_req.SensorPosXYZ = [0, 0, 0.1]

`VisionSensorReqNew` Class¶

A C++-style structure used to send requests and configure vision sensor parameters to the UE4 simulation environment; used in RflySim drone vision simulation to define vision sensor configuration information.

`init()`¶

Function Description: Initializes a VisionSensorReqNew instance object.
Parameters (Args): - None
Return Value (Returns):

VisionSensorReqNew instance object

Exceptions (Raises): - None

`imuDataCopter` Class¶

A data structure used to receive IMU sensor data from CopterSim, storing checksum, message sequence number, timestamp, accelerometer data, and gyroscope data.

`init(imu_name, node=None)`¶

Function Description: Initializes an imuDataCopter instance, setting the IMU data ROS topic name and ROS node.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`imu_name`	`str`	No	`"/rflysim/imu"`	ROS topic name for IMU data
`node`	`Any`	No	`None`	ROS node object used to publish IMU data

Return Value (Returns):

imuDataCopter instance object

Exceptions (Raises):

None

`AlignTime(img_time)`¶

Function Description: Aligns the IMU data timestamp with the image timestamp to match IMU data at the corresponding moment.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`img_time`	`double`	Yes	-	Image timestamp to be aligned

Return Value (Returns):

Aligned and matched IMU data

Exceptions (Raises):

None

`Imu2ros(node=None)`¶

Function Description: Converts the current IMU data into ROS format and publishes it.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`node`	`Any`	No	`None`	ROS node object used to publish IMU data; if None, uses the node set during initialization

Return Value (Returns):

None

Exceptions (Raises):

None

Example:

from RflySimSDK.vision import imuDataCopter
# Initialize IMU data reception object
imu = imuDataCopter()
# Align with image timestamp 123.45 to obtain corresponding IMU data
aligned_imu = imu.AlignTime(123.45)

`DistanceSensor` Class¶

A distance sensor functional class used to implement distance sensor-related simulation logic in RflySim UAV simulation.

`init()`¶

Function Description: Initializes a DistanceSensor class instance
Parameters (Args):
None
Return Value (Returns):

DistanceSensor instance object

Exceptions (Raises):
None

`SensorReqCopterSim` Class¶

This is a structure class for sending sensor data requests to UE4, corresponding to the C++ structure definition, primarily used to pass sensor request configuration information in RflySim UAV simulation.

`init()`¶

Function Description: Initializes an empty sensor request structure instance
Parameters (Args):
None
Return Value (Returns):

SensorReqCopterSim instance object

Exceptions (Raises):
None

`VisionCaptureApi` Class¶

This is the API class for the Python side to acquire images from UE4.

`init(ip='127.0.0.1')`¶

Function Description: Initializes the visual image capture API instance, establishing a connection with the UE4 image service.
Parameters (Args):

Parameter Name	Type	Default	Description
`ip`	`str`	`127.0.0.1`	IP address of the UE4 image service; use the default local loopback address for local operation

Return Value (Returns): None
Exceptions (Raises): None

`euler2quat(r, p, y, copter, coordinate_frame="global")`¶

Function Description: Converts Euler angles to quaternions, supporting conversion under different vehicles and coordinate systems.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`r`	`Any`	Yes	None	Roll angle Euler component
`p`	`Any`	Yes	None	Pitch angle Euler component
`y`	`Any`	Yes	None	Yaw angle Euler component
`copter`	`str`	Yes	None	Vehicle type identifier
`coordinate_frame`	`Any`	No	`"global"`	Coordinate frame type; defaults to global coordinate frame

Return Value (Returns): The converted quaternion
Exceptions (Raises): None

`addVisSensor(vsr=VisionSensorReq())`¶

Function Description: Adds a new legacy VisionSensorReq visual sensor request structure to the visual sensor list
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`vsr`	`Any`	No	`VisionSensorReq()`	Legacy visual sensor request structure to be added

Return Value (Returns): None
Exceptions (Raises): Exception — thrown if addition fails

`addVisSensor(vsr=VisionSensorReqNew())`¶

Function Description: Adds a new updated VisionSensorReqNew visual sensor request structure to the visual sensor list
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`vsr`	`Any`	No	`VisionSensorReqNew()`	Updated visual sensor request structure to be added

Return Value (Returns): None
Exceptions (Raises): Exception — thrown if addition fails

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

Function Description: Sends a SensorReqCopterSim-type UDP request message to CopterSim to request sensor data; the target CopterSim instance is specified via copterID.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`srcs`	`Any`	No	`SensorReqCopterSim()`	Sensor request structure
`copterID`	`Any`	No	1	ID of the target vehicle; specifies the index of the CopterSim instance to request data from
`IP`	`Any`	No	`"127.0.0.1"`	IP address of the host where the target CopterSim is running

Return Value (Returns): None
Exceptions (Raises): None

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)`¶

Function Description: Sends an IMU data request command to CopterSim; this function initializes a thread to listen for incoming IMU data.
The actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`copterID`	`Any`	No	1	ID of the target vehicle
`IP`	`Any`	No	`"127.0.0.1"`	IP address of the target host to which the request is sent
`freq`	`Any`	No	200	Requested IMU data transmission frequency

Return Value (Returns): None
Exceptions (Raises): None

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)`¶

Function Description: Sends an IMU data request command to CopterSim. The actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`copterID`	`Any`	No	1	ID of the target vehicle
`IP`	`Any`	No	`"127.0.0.1"`	IP address of the target host to which the request is sent
`freq`	`Any`	No	200	Requested IMU data transmission frequency

Return Value (Returns): None
Exceptions (Raises): None

`sendImuReqServe(copterID=1)`¶

Function Description: Sends an IMU data request command to CopterSim; this function initializes a thread to listen for and receive IMU data, where the actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`copterID`	`Any`	No	1	ID of the target vehicle

Return Value (Returns): None
Exceptions (Raises): None

`getIMUDataLoop(copterID)`¶

Function Description: Cyclically retrieves IMU data for a specified vehicle, serving as the worker function for the IMU listening thread.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`copterID`	`Any`	Yes	None	ID of the vehicle whose data is to be retrieved

Return Value (Returns): None
Exceptions (Raises): None

`get_all_ip()`¶

Function Description: Retrieves all available network interface IP addresses on the local machine.
Parameters (Args): None
Return Value (Returns): A list of all IP addresses on the local machine
Exceptions (Raises): None

`isIpLocal(IP)`¶

Function Description: Determines whether a given IP address is a local machine IP address.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`IP`	`Any`	Yes	None	The IP address to be checked

Return Value (Returns): bool — returns True if the IP is local, otherwise False
Exceptions (Raises): None

`StartTimeStmplisten()`¶

Function Description: Starts a listening thread to monitor port 20005 for Rfly time stamps corresponding to various vehicle IDs.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`getTimeStmp(CopterID=1)`¶

Function Description: Retrieves the latest Rfly time stamp corresponding to a specified vehicle ID.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`CopterID`	`Any`	No	1	ID of the target vehicle

Return Value (Returns): The latest Rfly time stamp for the specified vehicle
Exceptions (Raises): None

`endTimeStmplisten()`¶

Function Description: Terminates the time stamp listening thread, stopping the reception of time stamp data.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`TimeStmploop()`¶

Function Description: The worker function of the time stamp listening thread, continuously receiving and processing time stamp data sent by CopterSim.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

Function Description: Sends a visual sensor image update request to UE4.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`vs`	`Any`	No	`VisionSensorReq()`	Visual sensor request structure
`windID`	`Any`	No	0	Window ID identifier
`IP`	`Any`	No	""	IP address of the host running the target UE4; if empty, automatically determined

Return Value (Returns): None
Exceptions (Raises): Exception — thrown when the request fails to send

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

Function Description: Updates the visual sensor configuration for a specified RflySim3D window.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`vs`	-	No	`VisionSensorReqNew()`	Visual sensor request configuration object
`windID`	-	No	`0`	RflySim3D window index
`IP`	-	No	`""`	Target IP address; uses default if empty

Return Value (Returns): No return value
Exceptions (Raises): None

`sendUE4Cmd(cmd, windowID=-1)`¶

Function Description: Sends a custom command to UE4.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`cmd`	-	Yes	None	Command content to be sent
`windowID`	-	No	`-1`	Target RflySim3D window index; `-1` indicates sending to all windows

Return Value (Returns): No return value
Exceptions (Raises): None

`sendReqToUE4(windID=0, IP="")`¶

Function Description: Sends the visual sensor list to RflySim3D to request corresponding images; the target RflySim3D window index is specified via windID.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`windID`	-	No	`0`	Target RflySim3D window index
`IP`	-	No	`""`	Target IP address; uses default if empty

Return Value (Returns): No return value
Exceptions (Raises): None

`img_udp_thrdNew(udpSok, idx, typeID)`¶

Function Description: Worker function for the UDP image reception thread, handling UDP image reception for a specified visual sensor.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`udpSok`	-	Yes	None	UDP socket object
`idx`	-	Yes	None	Visual sensor index
`typeID`	-	Yes	None	Sensor type ID

Return Value (Returns): No return value
Exceptions (Raises): None

`img_mem_thrd(idxList)`¶

Function Description: Worker function for the shared memory image reception thread, handling image reception for visual sensors in the specified index list.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`idxList`	-	Yes	None	List of visual sensor indices to receive

Return Value (Returns): No return value
Exceptions (Raises): None

`startImgCap(isRemoteSend=False)`¶

Function Description: Starts the loop to receive images from UE4. When isRemoteSend is True, images obtained from memory are forwarded to the UDP port.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`isRemoteSend`	-	No	`False`	Whether to enable remote image forwarding; when enabled, images are forwarded from shared memory to the UDP port

Return Value (Returns): None
Exceptions (Raises): None

`sendImgUDPNew(idx)`¶

Function Description: Sends the image of the specified visual sensor index via the new UDP protocol.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`idx`	-	Yes	None	Visual sensor index

Return Value (Returns): No return value
Exceptions (Raises): None

`sendImgBuffer(idx, data)`¶

Function Description: Sends the image buffer data of the specified visual sensor index.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`idx`	-	Yes	None	Visual sensor index
`data`	-	Yes	None	Image data buffer to be sent

Return Value (Returns): No return value
Exceptions (Raises): None

`jsonLoad(ChangeMode=-1, jsonPath="")`¶

Function Description: Loads a configuration JSON file and creates a camera list for image acquisition. If ChangeMode >= 0, the first item SendProtocol[0] is set to ChangeMode to modify the image transmission mode.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`ChangeMode`	-	No	`-1`	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration
`jsonPath`	-	No	`""`	Path to the configuration JSON file; uses default path if empty

Return Value (Returns): No return value
Exceptions (Raises): None

`dictLoad(jsData, ChangeMode=-1)`¶

Function Description: Wrapper for the jsonLoad method, enabling direct reading of sensor data stored in memory as a dictionary. The data structure matches that of the JSON file used in jsonLoad.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`jsData`	-	Yes	None	Dictionary of sensor configuration data stored in memory
`ChangeMode`	-	No	`-1`	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration

Return Value (Returns): No return value
Exceptions (Raises): None

`stopRun()`¶

Function Description: Stops the image acquisition runtime loop and terminates all image-reception threads.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`SensorPreview(seq_id=-1)`¶

Function Description: Previews images captured by the visual sensor with the specified sequence ID.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`seq_id`	-	No	`-1`	Sequence ID of the visual sensor to preview; `-1` indicates previewing all sensors

Return Value (Returns): None
Exceptions (Raises): None

Advanced Usage Example¶

Demonstrates complex composite scenarios (e.g., multi-type collaboration, asynchronous control, batch operations)

from RflySimSDK.vision import VisionCaptureApi, Queue, imuDataCopter
import threading

# Initialize a queue to buffer sensor data, enabling asynchronous data reception
imu_data_queue = Queue()
vision_capture = VisionCaptureApi()

# Batch add 5 heterogeneous visual sensors to the simulation environment
for sensor_id in range(5):
    # Customize installation poses for different sensors, automatically converting to quaternions
    roll, pitch, yaw = 0.1 * sensor_id, 0.05, -0.02 * sensor_id
    quat = vision_capture.euler2quat(roll, pitch, yaw)
    vision_capture.addVisSensor(sensor_id=sensor_id, position=[0.1 * sensor_id, 0, 0.2], quaternion=quat)

# Start an asynchronous IMU data acquisition thread; after time alignment, store data in a queue
def get_imu_async():
    imu_handler = imuDataCopter()
    while True:
        raw_imu = vision_capture.getIMUDataLoop()
        if raw_imu is not None:
            # Align IMU timestamps with visual frame timestamps
            aligned_imu = imu_handler.AlignTime(raw_imu)
            if not imu_data_queue.is_empty() and imu_data_queue.size() > 10:
                imu_data_queue.dequeue()
            imu_data_queue.enqueue(aligned_imu)

# Start asynchronous thread for data processing; main thread synchronously requests visual frames for SLAM computation
threading.Thread(target=get_imu_async, daemon=True).start()
vision_capture.sendReqToCopterSim()

Notes and Pitfall Avoidance Guide¶

Duplicate Visual Sensor Addition Issue: When calling addVisSensor multiple times, ensure the input sensor_id is globally unique; duplicate IDs will cause the simulation side to overwrite sensors, resulting in erroneous visual data retrieval.
Asynchronous Data Buffer Overflow: When using a custom queue to cache IMU or visual data, periodically clean up old data; prolonged operation will cause the queue to accumulate and consume excessive memory. This can be mitigated by checking the queue size and popping the oldest data from the front once it exceeds a threshold.
Cross-Host IP Connection Check: Before establishing a connection, call isIpLocal to detect IP properties. If the simulation side runs on a non-local host, ensure the IP port permissions are open; otherwise, the default port may be blocked by the remote firewall, causing request timeouts.
Prerequisites for Time Alignment: Before calling imuDataCopter.AlignTime, time stamp initialization must be completed first. Failure to call RflyTimeStmp.Update to update the reference time stamp will result in alignment deviations, causing SLAM pose jumps.

Changelog¶

2026-03-03: feat: SDK adds IP handling mechanism, compatible with cloud deployment from local version
2026-02-02: feat: Add odom publishing for depth-to-point-cloud conversion
2026-02-02: fix: Optimize protocol
2026-02-01: fix: Comment out path printing
2026-01-31: fix: Update depth-to-point-cloud protocol
2026-01-31: fix: Update communication protocol for depth-to-point-cloud
2026-01-23: Align timestamps for /Depth_Cloud topic
2026-01-22: Merge branch 'master' of http://rflysim.synology.me:8418/FeisiLab/RflySimSDK
2026-01-22: Change IMU coordinate frame orientation for ROS output to ensure consistent FLU frame output; previously, conversion was FRD → FLU, but the platform’s IMU angular velocity conforms to FRD, while linear acceleration does not conform to FRD but rather BLD
2025-12-12: fix: Optimize image timestamp synchronization mechanism, resolve image timestamp desynchronization bug on Windows
2025-12-08: Change platform odom data translation in local frame under mode 0 from FRD → FLU
2025-12-07: fix: Add relative position transmission data for LiDAR
2025-12-04: fix: Fix point cloud sensor issue
2025-09-30: Resolve fisheye camera bug under UDP mode
2025-09-29: ROS2 depth-to-point-cloud seq variable has been removed; interface needs adaptation

Parameter Name	Type	Required	Default Value	Description
r	`Any`	Yes	None	Roll angle Euler component
p	`Any`	Yes	None	Pitch angle Euler component
y	`Any`	Yes	None	Yaw angle Euler component
copter	`str`	Yes	None	Vehicle type identifier
coordinate_frame	`Any`	No	`"global"`	Coordinate frame type; defaults to global coordinate frame

Return Value (Returns): The converted quaternion
Exceptions (Raises): None

`addVisSensor(vsr=VisionSensorReq())`¶

Function Description: Adds a new legacy VisionSensorReq visual sensor request structure to the visual sensor list
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vsr	`Any`	No	`VisionSensorReq()`	Legacy visual sensor request structure to be added

Return Value (Returns): None
Exceptions (Raises): Exception — thrown if addition fails

`addVisSensor(vsr=VisionSensorReqNew())`¶

Function Description: Adds a new updated VisionSensorReqNew visual sensor request structure to the visual sensor list
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vsr	`Any`	No	`VisionSensorReqNew()`	Updated visual sensor request structure to be added

Return Value (Returns): None
Exceptions (Raises): Exception — thrown if addition fails

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

Function Description: Sends a SensorReqCopterSim-type UDP request message to CopterSim to request sensor data; the target CopterSim instance is specified via copterID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
srcs	`Any`	No	`SensorReqCopterSim()`	Sensor request structure
copterID	`Any`	No	1	ID of the target vehicle; specifies the index of the CopterSim instance to request data from
IP	`Any`	No	`"127.0.0.1"`	IP address of the host where the target CopterSim is running

Return Value (Returns): None
Exceptions (Raises): None

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)`¶

Function Description: Sends an IMU data request command to CopterSim; this function initializes a thread to listen for incoming IMU data.
The actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	No	1	ID of the target vehicle
IP	`Any`	No	`"127.0.0.1"`	IP address of the target host to which the request is sent
freq	`Any`	No	200	Requested IMU data transmission frequency

Return Value (Returns): None
Exceptions (Raises): None

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)`¶

Function Description: Sends an IMU data request command to CopterSim. The actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	No	1	ID of the target vehicle
IP	`Any`	No	`"127.0.0.1"`	IP address of the target host to which the request is sent
freq	`Any`	No	200	Requested IMU data transmission frequency

Return Value (Returns): None
Exceptions (Raises): None

`sendImuReqServe(copterID=1)`¶

Function Description: Sends an IMU data request command to CopterSim; this function initializes a thread to listen for and receive IMU data, where the actual port = base port + vehicle ID - 1.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	No	1	ID of the target vehicle

Return Value (Returns): None
Exceptions (Raises): None

`getIMUDataLoop(copterID)`¶

Function Description: Cyclically retrieves IMU data for a specified vehicle, serving as the worker function for the IMU listening thread.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	Yes	None	ID of the vehicle whose data is to be retrieved

Return Value (Returns): None
Exceptions (Raises): None

`get_all_ip()`¶

Function Description: Retrieves all available network interface IP addresses on the local machine.
Parameters (Args): None
Return Value (Returns): A list of all IP addresses on the local machine
Exceptions (Raises): None

`isIpLocal(IP)`¶

Function Description: Determines whether a given IP address is a local machine IP address.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
IP	`Any`	Yes	None	The IP address to be checked

Return Value (Returns): bool — returns True if the IP is local, otherwise False
Exceptions (Raises): None

`StartTimeStmplisten()`¶

Function Description: Starts a listening thread to monitor port 20005 for Rfly time stamps corresponding to various vehicle IDs.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`getTimeStmp(CopterID=1)`¶

Function Description: Retrieves the latest Rfly time stamp corresponding to a specified vehicle ID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
CopterID	`Any`	No	1	ID of the target vehicle

Return Value (Returns): The latest Rfly time stamp for the specified vehicle
Exceptions (Raises): None

`endTimeStmplisten()`¶

Function Description: Terminates the time stamp listening thread, stopping the reception of time stamp data.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`TimeStmploop()`¶

Function Description: The worker function of the time stamp listening thread, continuously receiving and processing time stamp data sent by CopterSim.
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

Function Description: Sends a visual sensor image update request to UE4.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vs	`Any`	No	`VisionSensorReq()`	Visual sensor request structure
windID	`Any`	No	0	Window ID identifier
IP	`Any`	No	""	IP address of the host running the target UE4; if empty, automatically determined

Return Value (Returns): None
Exceptions (Raises): Exception — thrown when the request fails to send

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

Function Description: Updates the visual sensor configuration for a specified RflySim3D window.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vs	-	No	`VisionSensorReqNew()`	Visual sensor request configuration object
windID	-	No	`0`	RflySim3D window index
IP	-	No	`""`	Target IP address; uses default if empty

Return Value (Returns): No return value
Exceptions (Raises): Exception — thrown if an error occurs during sending

`sendUE4Cmd(cmd, windowID=-1)`¶

Function Description: Sends a custom command to UE4.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
cmd	-	Yes	None	Command content to be sent
windowID	-	No	`-1`	Target RflySim3D window index; `-1` indicates sending to all windows

Return Value (Returns): No return value
Exceptions (Raises): None

`sendReqToUE4(windID=0, IP="")`¶

Function Description: Sends the visual sensor list to RflySim3D to request corresponding images; the target RflySim3D window index is specified via windID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
windID	-	No	`0`	Target RflySim3D window index
IP	-	No	`""`	Target IP address; uses default if empty

Return Value (Returns): No return value
Exceptions (Raises): None

`img_udp_thrdNew(udpSok, idx, typeID)`¶

Function Description: Worker function for the UDP image reception thread, handling UDP image reception for a specified visual sensor.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
udpSok	-	Yes	None	UDP socket object
idx	-	Yes	None	Visual sensor index
typeID	-	Yes	None	Sensor type ID

Return Value (Returns): No return value
Exceptions (Raises): None

`img_mem_thrd(idxList)`¶

Function Description: Worker function for the shared memory image reception thread, handling image reception for visual sensors in the specified index list.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
idxList	-	Yes	None	List of visual sensor indices to receive

Return Value (Returns): No return value
Exceptions (Raises): None

`startImgCap(isRemoteSend=False)`¶

Function Description: Starts the loop to receive images from UE4. When isRemoteSend is True, images obtained from memory are forwarded to the UDP port.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
isRemoteSend	-	No	`False`	Whether to enable remote image forwarding; when enabled, images are forwarded from shared memory to the UDP port

Return Value (Returns): None
Exceptions (Raises): None

Return Value (Returns): No return value
Exceptions (Raises): None

`sendImgUDPNew(idx)`¶

Function Description: Sends the image of the specified visual sensor index via the new UDP protocol.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
`idx`	-	Yes	None	Visual sensor index

Return Value (Returns): No return value
Exceptions (Raises): None

`sendImgBuffer(idx, data)`¶

Function Description: Sends the image buffer data of the specified visual sensor index.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
`idx`	-	Yes	None	Visual sensor index
`data`	-	Yes	None	Image data buffer to be sent

Return Value (Returns): No return value
Exceptions (Raises): None

`jsonLoad(ChangeMode=-1, jsonPath="")`¶

Function Description: Loads a configuration JSON file and creates a camera list for image acquisition. If ChangeMode >= 0, the first item SendProtocol[0] is set to ChangeMode to modify the image transmission mode.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
`ChangeMode`	-	No	`-1`	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration
`jsonPath`	-	No	`""`	Path to the configuration JSON file; uses default path if empty

Return Value (Returns): No return value
Exceptions (Raises): None

`dictLoad(jsData, ChangeMode=-1)`¶

Function Description: Wrapper for the jsonLoad method, enabling direct reading of sensor data stored in memory as a dictionary. The data structure matches that of the JSON file used in jsonLoad.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
`jsData`	-	Yes	None	Dictionary of sensor configuration data stored in memory
`ChangeMode`	-	No	`-1`	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration

Return Value (Returns): No return value
Exceptions (Raises): None

`stopRun()`¶

Function Description: Stops the image acquisition runtime loop and terminates all image-reception threads.
Parameters (Args): None
Return Value (Returns): No return value
Exceptions (Raises): None

`SensorPreview(seq_id=-1)`¶

Function Description: Previews images captured by the visual sensor with the specified sequence ID.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
`seq_id`	-	No	`-1`	Sequence ID of the visual sensor to preview; `-1` indicates previewing all sensors

Return Value (Returns): No return value
Exceptions (Raises): None

Advanced Usage Example¶

The following example requests LiDAR point cloud data and continuously updates the new point cloud in vis.Img[0] to an Open3D window.

Reference example: [RflySim installation path]\RflySimAPIs\8.RflySimVision\0.ApiExps\1-UsageAPI\3.PointCloudAPI\2.LidarAPIPointCloudDemo

import sys
import time

import Open3DShow
import VisionCaptureApi


show3d = Open3DShow.Open3DShow()
vis = VisionCaptureApi.VisionCaptureApi()

vis.jsonLoad()
if not vis.sendReqToUE4():
    sys.exit(0)
vis.startImgCap()
show3d.CreatShow(0)

lastTime = time.time()
while True:
    lastTime += 1 / 10.0
    sleepTime = lastTime - time.time()
    if sleepTime > 0:
        time.sleep(sleepTime)
    else:
        lastTime = time.time()

    if vis.hasData[0]:
        show3d.UpdateShow(vis.Img[0])
        vis.hasData[0] = False

Notes and Pitfall Avoidance Guide¶

Duplicate Visual Sensor Addition Issue: When calling addVisSensor multiple times, ensure the input sensor_id is globally unique; duplicate IDs will cause the simulation side to overwrite sensors, resulting in erroneous visual data retrieval.
Asynchronous Data Buffer Overflow: When using a custom queue to cache IMU or visual data, periodically clean up old data; prolonged operation will cause the queue to accumulate and consume excessive memory. This can be mitigated by checking the queue size and popping the oldest data from the front once it exceeds a threshold.
Cross-Host IP Connection Check: Before establishing a connection, call isIpLocal to detect IP properties. If the simulation side runs on a non-local host, ensure the IP port permissions are open; otherwise, the default port may be blocked by the remote firewall, causing request timeouts.
Prerequisites for Time Alignment: Before calling imuDataCopter.AlignTime, time stamp initialization must be completed first. Failure to call RflyTimeStmp.Update to update the reference time stamp will result in alignment deviations, causing SLAM pose jumps.

Changelog¶

2026-03-03: feat: SDK adds IP handling mechanism, compatible with cloud deployment from local version
2026-02-02: feat: Add odom publishing for depth-to-point-cloud conversion
2026-02-02: fix: Optimize protocol
2026-02-01: fix: Comment out path printing
2026-01-31: fix: Update depth-to-point-cloud protocol
2026-01-31: fix: Update communication protocol for depth-to-point-cloud
2026-01-23: Align timestamps for /Depth_Cloud topic
2026-01-22: Merge branch 'master' of http://rflysim.synology.me:8418/FeisiLab/RflySimSDK
2026-01-22: Change IMU coordinate frame orientation for ROS output to ensure consistent FLU frame output; previously, conversion was FRD → FLU, but the platform’s IMU angular velocity conforms to FRD, while linear acceleration does not conform to FRD but rather BLD
2025-12-12: fix: Optimize image timestamp synchronization mechanism, resolve image timestamp desynchronization bug on Windows
2025-12-08: Change platform odom data translation in local frame under mode 0 from FRD → FLU
2025-12-07: fix: Add relative position transmission data for LiDAR
2025-12-04: fix: Fix point cloud sensor issue
2025-09-30: Resolve fisheye camera bug under UDP mode
2025-09-29: ROS2 depth-to-point-cloud seq variable has been removed; interface needs adaptation

VisionCaptureApi Interface Documentation¶

Introduction¶

Quick Start¶

Environment and Dependencies¶

Core Interface Description¶

Global Constants and Enumerations¶

Standalone Constants¶

Global/Standalone Functions¶

Queue Class¶

__init__()¶

enqueue(item)¶

dequeue()¶

is_empty()¶

size()¶

RflyTimeStmp Class¶

__init__(iv)¶

Update(iv)¶

VisionSensorReq Class¶

__init__()¶

VisionSensorReqNew Class¶

__init__()¶

imuDataCopter Class¶

__init__(imu_name, node=None)¶

AlignTime(img_time)¶

Imu2ros(node=None)¶

DistanceSensor Class¶

__init__()¶

SensorReqCopterSim Class¶

__init__()¶

VisionCaptureApi Class¶

__init__(ip='127.0.0.1')¶

euler2quat(r, p, y, copter, coordinate_frame="global")¶

addVisSensor(vsr=VisionSensorReq())¶

addVisSensor(vsr=VisionSensorReqNew())¶

sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")¶

sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)¶

sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)¶

sendImuReqServe(copterID=1)¶

getIMUDataLoop(copterID)¶

get_all_ip()¶

isIpLocal(IP)¶

StartTimeStmplisten()¶

getTimeStmp(CopterID=1)¶

endTimeStmplisten()¶

TimeStmploop()¶

sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")¶

sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")¶

sendUE4Cmd(cmd, windowID=-1)¶

sendReqToUE4(windID=0, IP="")¶

img_udp_thrdNew(udpSok, idx, typeID)¶

img_mem_thrd(idxList)¶

startImgCap(isRemoteSend=False)¶

sendImgUDPNew(idx)¶

sendImgBuffer(idx, data)¶

jsonLoad(ChangeMode=-1, jsonPath="")¶

dictLoad(jsData, ChangeMode=-1)¶

stopRun()¶

SensorPreview(seq_id=-1)¶

Advanced Usage Example¶

Notes and Pitfall Avoidance Guide¶

Changelog¶

addVisSensor(vsr=VisionSensorReq())¶

addVisSensor(vsr=VisionSensorReqNew())¶

sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")¶

sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)¶

sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)¶

sendImuReqServe(copterID=1)¶

getIMUDataLoop(copterID)¶

get_all_ip()¶

isIpLocal(IP)¶

StartTimeStmplisten()¶

getTimeStmp(CopterID=1)¶

endTimeStmplisten()¶

TimeStmploop()¶

sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")¶

sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")¶

sendUE4Cmd(cmd, windowID=-1)¶

sendReqToUE4(windID=0, IP="")¶

img_udp_thrdNew(udpSok, idx, typeID)¶

img_mem_thrd(idxList)¶

`Queue` Class¶

`init()`¶

`enqueue(item)`¶

`dequeue()`¶

`is_empty()`¶

`size()`¶

`RflyTimeStmp` Class¶

`init(iv)`¶

`Update(iv)`¶

`VisionSensorReq` Class¶

`init()`¶

`VisionSensorReqNew` Class¶

`init()`¶

`imuDataCopter` Class¶

`init(imu_name, node=None)`¶

`AlignTime(img_time)`¶

`Imu2ros(node=None)`¶

`DistanceSensor` Class¶

`init()`¶

`SensorReqCopterSim` Class¶

`init()`¶

`VisionCaptureApi` Class¶

`init(ip='127.0.0.1')`¶

`euler2quat(r, p, y, copter, coordinate_frame="global")`¶

`addVisSensor(vsr=VisionSensorReq())`¶

`addVisSensor(vsr=VisionSensorReqNew())`¶

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)`¶

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)`¶

`sendImuReqServe(copterID=1)`¶

`getIMUDataLoop(copterID)`¶

`get_all_ip()`¶

`isIpLocal(IP)`¶

`StartTimeStmplisten()`¶

`getTimeStmp(CopterID=1)`¶

`endTimeStmplisten()`¶

`TimeStmploop()`¶

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

`sendUE4Cmd(cmd, windowID=-1)`¶

`sendReqToUE4(windID=0, IP="")`¶

`img_udp_thrdNew(udpSok, idx, typeID)`¶

`img_mem_thrd(idxList)`¶

`startImgCap(isRemoteSend=False)`¶

`sendImgUDPNew(idx)`¶

`sendImgBuffer(idx, data)`¶

`jsonLoad(ChangeMode=-1, jsonPath="")`¶

`dictLoad(jsData, ChangeMode=-1)`¶

`stopRun()`¶

`SensorPreview(seq_id=-1)`¶

`addVisSensor(vsr=VisionSensorReq())`¶

`addVisSensor(vsr=VisionSensorReqNew())`¶

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)`¶

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)`¶

`sendImuReqServe(copterID=1)`¶

`getIMUDataLoop(copterID)`¶

`get_all_ip()`¶

`isIpLocal(IP)`¶

`StartTimeStmplisten()`¶

`getTimeStmp(CopterID=1)`¶

`endTimeStmplisten()`¶

`TimeStmploop()`¶

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

`sendUE4Cmd(cmd, windowID=-1)`¶

`sendReqToUE4(windID=0, IP="")`¶

`img_udp_thrdNew(udpSok, idx, typeID)`¶

`img_mem_thrd(idxList)`¶

`startImgCap(isRemoteSend=False)`¶

`sendImgUDPNew(idx)`¶

`sendImgBuffer(idx, data)`¶

`jsonLoad(ChangeMode=-1, jsonPath="")`¶

`dictLoad(jsData, ChangeMode=-1)`¶

`stopRun()`¶

`SensorPreview(seq_id=-1)`¶