VisionCaptureApi Interface Documentation¶

Introduction¶

Overview: This document defines the data structures and API classes required by the visual perception module of the RflySim UAV simulation platform, providing foundational interactive support for Python-side requests to the UE4 simulation environment for camera images and various sensor data.

In RflySim simulation, visual tasks are central to the development of intelligent algorithms such as UAV autonomous navigation and target detection, requiring parameter exchange and data transmission between the Python and UE4 sides. This module defines structures for sending camera parameter settings and image requests to UE4, as well as encapsulation structures for various perception data—including IMU data and distance sensor data—from CopterSim. It also provides the core VisionCaptureApi class, enabling Python-side acquisition of simulated images from the UE4 simulation environment. It is suitable for developing vision-based UAV simulation algorithms, offering a unified interface specification for visual perception data interaction in simulation environments, supporting monocular and multi-camera simulated data acquisition. Combined with IMU and distance sensor information, it enables multi-sensor fusion simulation tasks.

Quick Start¶

The following example loads sensor configuration from Config.json, sends an image request to RflySim3D, and displays the received images from each channel.

Reference example: [RflySim installation path]\RflySimAPIs\8.RflySimVision\0.ApiExps\1-UsageAPI\0.ConfigJsonAPI

import sys

import cv2
import VisionCaptureApi


vis = VisionCaptureApi.VisionCaptureApi()
vis.jsonLoad()

for sensor in vis.VisSensor:
    print("SeqID:", sensor.SeqID)
    print("SendProtocol:", sensor.SendProtocol)

if not vis.sendReqToUE4():
    sys.exit(0)
vis.startImgCap()

while True:
    for index in range(len(vis.hasData)):
        if vis.hasData[index]:
            cv2.imshow("Img" + str(index), vis.Img[index])
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

Environment and Dependencies¶

Python Environment: >= 3.8.10
Dependencies: copy, ctrl.IpManager, cv2, email, ipaddress, json, math, mmap, numpy, os, platform, psutil, re, socket, struct, subprocess, sys, threading, time, warnings
Optional Dependency: Open3DShow is only attempted for import on Windows or WSL; when unavailable, point clouds can still be received, but SensorPreview() will not create a point cloud display window.
Prerequisites: Before calling this interface, ensure that the RflySimSDK vision module has been correctly imported.

Core Interface Description¶

The module VisionCaptureApi.py includes configuration variables, helper functions, and the core business class.

Global Constants and Enumerations¶

This section lists all globally accessible constants and enumeration definitions directly referenceable within the module.

Standalone Constants¶

Name	Default Value	Description
`SENSOR_REQ_STRUCT`	`<4H4B6f`	Little-endian, no-padding protocol for CopterSim sensor requests, 36 bytes in total.
`IMU_DATA_STRUCT`	`<iid6f`	IMU return protocol, 40 bytes in total.
`ODOM_DATA_STRUCT`	`<IIdIHH3d4f3f3f`	Odom return protocol, 88 bytes in total, with quaternion order `w, x, y, z`.
`IMU_DATA_CHECKSUM`	`1234567898`	IMU packet checksum.
`ODOM_DATA_CHECKSUM`	`1234567888`	Odom packet checksum.
`ODOM_FRAME_NED_FRD`	`1`	Odom linear data uses the NED/FRD coordinate convention.
`ODOM_POSITION_VALID`	`1 << 0`	Position field valid bit.
`ODOM_ORIENTATION_VALID`	`1 << 1`	Orientation field valid bit.
`ODOM_LINEAR_VELOCITY_VALID`	`1 << 2`	Linear velocity field valid bit.
`ODOM_ANGULAR_VELOCITY_VALID`	`1 << 3`	Angular velocity field valid bit.
`ODOM_DISCONTINUITY`	`1 << 5`	Odom discontinuity flag; when set, resets the initial transform for that aircraft.
`udpDecodeMode`	`2`	`1` means packet reception and decoding in the same thread; `2` means separate threads for packet reception and decoding.
`isPrintUdpStat`	`0`	Whether to print UDP packet reception and frame loss statistics.
`isUseRawUdpTimestamp`	`False`	Whether ROS messages directly use the simulation-relative timestamp within the UDP packet.
`publishCopterSimClock`	`False`	Whether a single ROS1 simulation publishes the CopterSim `/clock`; when enabled, requires `isUseRawUdpTimestamp=True`.

Global/Standalone Functions¶

None

`Queue` Class¶

Provides a basic queue data structure for storing and managing elements in a first-in-first-out (FIFO) manner.

`init()`¶

Function Description: Initializes an empty queue instance Parameters (Args): None Returns:

Queue instance object

Exceptions (Raises): None

`enqueue(item)`¶

Function Description: Adds an element to the tail of the queue Parameters (Args):

Parameter Name	Type	Required	Default	Description
`item`	`Any type`	Yes	-	Element to be added to the queue

Returns:

None

Exceptions (Raises): None

`dequeue()`¶

Function Description: Removes and returns the element at the head of the queue Parameters (Args): None Returns:

Element at the head of the queue

Exceptions (Raises): None

`is_empty()`¶

Function Description: Determines whether the current queue is empty Parameters (Args): None Returns:

bool: Returns True if the queue is empty, otherwise returns False

Exceptions (Raises): None

`size()`¶

Function Description: Retrieves the current number of elements stored in the queue Parameters (Args): None Returns:

int: Number of elements in the queue

Exceptions (Raises): None

Example:

from RflySimSDK.vision import Queue

# Create a queue and add elements
q = Queue()
q.enqueue("first frame")
q.enqueue("second frame")

# Check queue size
print(q.size())  # Outputs: 2

# Dequeue an element
item = q.dequeue()
print(item)  # Outputs: first frame

# Check if queue is empty
print(q.is_empty())  # Outputs: False

`RflyTimeStmp` Class¶

A utility class for managing timestamp data of drone vision data, capable of storing and updating timestamp values, used in conjunction with the vision capture module.

`init(iv)`¶

Function Description: Initializes an RflyTimeStmp instance and sets the initial timestamp value. Parameters (Args):

Parameter Name	Type	Required	Default	Description
`iv`	Any numeric type	Yes	-	Initial timestamp value

Return Value (Returns):

RflyTimeStmp instance object

Exceptions (Raises): None

`Update(iv)`¶

Function Description: Updates the stored timestamp value to the new input value. Parameters (Args):

Parameter Name	Type	Required	Default	Description
`iv`	Any numeric type	Yes	-	New timestamp value to be updated

Return Value (Returns): None Exceptions (Raises): None

Example:

from RflySimSDK.vision import RflyTimeStmp
# Initialize timestamp to 0
ts = RflyTimeStmp(0)
# Update timestamp to current simulation time 12.34
ts.Update(12.34)

`VisionSensorReq` Class¶

A C++ structure corresponding to a Python class used to request and configure camera parameters in the UE4 simulation environment; applicable for configuring vision sensor parameters in RflySim drone vision simulation.

`init()`¶

Function Description: Initializes a VisionSensorReq instance; all parameter members are created according to the default structure definition. Parameters (Args): None Return Value (Returns):

VisionSensorReq instance object

Exceptions (Raises): None

Example:

from RflySimSDK.vision import VisionSensorReq
# Create a vision sensor parameter request object
sensor_req = VisionSensorReq()
# Configure the target aircraft ID
sensor_req.TargetCopter = 1
# Set the camera mounting position
sensor_req.SensorPosXYZ = [0, 0, 0.1]

`VisionSensorReqNew` Class¶

A C++-style structure used to send requests and configure vision sensor parameters to the UE4 simulation environment; used in RflySim drone vision simulation to define vision sensor configuration information.

`init()`¶

Function Description: Initializes a VisionSensorReqNew instance object. Parameters (Args): None Return Value (Returns):

VisionSensorReqNew instance object

Exceptions (Raises): None

`imuDataCopter` Class¶

A data structure used to receive IMU sensor data from CopterSim, storing checksum, message sequence number, timestamp, accelerometer data, and gyroscope data.

`init(imu_name, node=None)`¶

Function Description: Initializes an imuDataCopter instance, setting the IMU data topic name and ROS node. Parameters (Args):

Parameter Name	Type	Required	Default	Description
`imu_name`	`str`	No	`"/rflysim/imu"`	ROS topic name for IMU data
`node`	`Any`	No	`None`	ROS node object used to publish IMU data

Return Value (Returns):

imuDataCopter instance object

Exceptions (Raises):

None

`AlignTime(img_time)`¶

Function Description: Aligns the IMU data timestamp with the image timestamp to match IMU data at the corresponding moment. Parameters (Args):

Parameter Name	Type	Required	Default	Description
`img_time`	`double`	Yes	-	Image timestamp to be aligned

Return Value (Returns):

Aligned and matched IMU data

Exceptions (Raises):

None

`Imu2ros(node=None)`¶

Function Description: Converts the current IMU data into ROS format and publishes it. Parameters (Args):

Parameter Name	Type	Required	Default	Description
`node`	`Any`	No	`None`	ROS node object used to publish IMU data; if None, uses the node set during initialization

Return Value (Returns):

None

Exceptions (Raises):

None

Example:

from RflySimSDK.vision import imuDataCopter
# Initialize the IMU data reception object
imu = imuDataCopter()
# Align with image timestamp 123.45 to obtain corresponding IMU data
aligned_imu = imu.AlignTime(123.45)

`imuDataCopterSnapshot` Class¶

A snapshot of one frame of IMU data stored per aircraft. The instance contains checksum, seq, simulation-relative time timestmp, three-axis acceleration acc, three-axis angular velocity rate, the aligned imuStmp, and the time reference rflyStartStmp. getImuData() returns a deep copy of this object; modifications by the caller will not affect the cache in the receiving thread.

`init()`¶

Function Description: Initializes numeric fields to 0, arrays to three floating-point zeros, and the checksum to IMU_DATA_CHECKSUM.

Parameters (Args): None

Return Value (Returns): An imuDataCopterSnapshot instance

`odomDataCopter` Class¶

One frame of CopterSim Odom data stored per aircraft, preserving the NED/FRD coordinate convention of the wire protocol. Main fields include seq, timestmp, copterID, frameType, flags, position[3], quaternion[4] (w, x, y, z), linearVelocity[3], angularVelocity[3], odomStmp, and rflyStartStmp.

`init()`¶

Function Description: Initializes the Odom snapshot with an identity quaternion and zero position and velocity; defaults to frameType=ODOM_FRAME_NED_FRD and checksum=ODOM_DATA_CHECKSUM.

Parameters (Args): None

Return Value (Returns): An odomDataCopter instance

`DistanceSensor` Class¶

A distance sensor functional class used to implement distance sensor-related simulation logic in RflySim UAV simulation.

`init()`¶

Function Description: Initializes a DistanceSensor class instance Parameters (Args): None Return Value (Returns):

DistanceSensor instance object

Exceptions (Raises): None

`OpticalFlowSensor` Class¶

TypeID 10 optical flow data container, storing time_usec, sensor_id, raw optical flow flow_x/flow_y, angular-velocity-compensated flow_comp_m_x/flow_comp_m_y, quality, ground distance ground_distance, and flow_rate_x/flow_rate_y.

`init()`¶

Function Description: Initializes all optical flow data fields to 0. The protocol range of quality is 0 (unavailable) to 255 (highest quality); a negative value of ground_distance indicates that the distance is unknown.

Parameters (Args): None

Return Value (Returns): An OpticalFlowSensor instance

`SensorReqCopterSim` Class¶

This is a structure class for sending sensor data requests to UE4, corresponding to the C++ structure definition, primarily used to pass sensor request configuration information in RflySim UAV simulation.

`init()`¶

Function Description: Initializes an empty sensor request structure instance Parameters (Args): None Return Value (Returns):

SensorReqCopterSim instance object

Exceptions (Raises): None

`VisionCaptureApi` Class¶

This is the API class for the Python side to acquire images from UE4.

`init(ip='127.0.0.1')`¶

Function Description: Initializes the visual image capture API instance, establishing a connection with the UE4 image service. Parameters (Args):

Parameter Name	Type	Default	Description
ip	`str`	`127.0.0.1`	IP address of the UE4 image service; use the default local loopback address for local operation

Return Value (Returns): None Exceptions (Raises): None

`euler2quat(r, p, y, copter, coordinate_frame="global")`¶

Function Description: Converts Euler angles to quaternions, supporting conversion under different vehicles and coordinate systems Parameters (Args):

Parameter Name	Type	Required	Default	Description
r	`Any`	Yes	None	Roll angle Euler component
p	`Any`	Yes	None	Pitch angle Euler component
y	`Any`	Yes	None	Yaw angle Euler component
copter	`str`	Yes	None	Vehicle type identifier
coordinate_frame	`Any`	No	"global"	Coordinate frame type; defaults to global coordinate frame

Return Value (Returns): The converted quaternion Exceptions (Raises): None

`addVisSensor(vsr=VisionSensorReq())`¶

Function Description: Adds a new legacy VisionSensorReq visual sensor request structure to the visual sensor list Parameters (Args):

Parameter Name	Type	Required	Default	Description
vsr	`Any`	No	`VisionSensorReq()`	Legacy visual sensor request structure to be added

Return Value (Returns): None Exceptions (Raises): Exception — thrown if addition fails

`addVisSensor(vsr=VisionSensorReqNew())`¶

Function Description: Adds a new updated VisionSensorReqNew visual sensor request structure to the visual sensor list Parameters (Args):

Parameter Name	Type	Required	Default	Description
vsr	`Any`	No	`VisionSensorReqNew()`	Updated visual sensor request structure to be added

Return Value (Returns): None Exceptions (Raises): Exception — thrown if addition fails

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

Function Description: Sends a SensorReqCopterSim-type UDP request message to CopterSim to request sensor data; the target CopterSim instance is specified via copterID Parameters (Args):

Parameter Name	Type	Required	Default	Description
srcs	`Any`	No	`SensorReqCopterSim()`	Sensor request structure
copterID	`Any`	No	1	ID of the target vehicle; specifies the index of the CopterSim instance to request data from
IP	`Any`	No	"127.0.0.1"	IP address of the host where the target CopterSim is running

Return Value (Returns): None Exceptions (Raises): None

`sendOdomReqCopterSim(copterID=1, IP="127.0.0.1", freq=100, port=None)`¶

Function Description: Starts or reuses the shared IMU/Odom UDP receiving thread for the specified aircraft, marks that aircraft as an independent Odom data source, and then sends an Odom request with sensorType=1 to CopterSim.

Parameter Name	Type	Default	Description
copterID	`int`	`1`	Target aircraft ID; must be greater than or equal to 1.
IP	`str`	`127.0.0.1`	Address of the host where CopterSim is running.
freq	`int`	`100`	Request frequency; allowed range 1 to 1000 Hz.
port	`int \| None`	`None`	Local return port; defaults to `31000 + copterID - 1`.

Return Value (Returns): None

Exceptions (Raises): Throws ValueError when a parameter is out of range or RemotSendIP is not a valid IPv4 address; throws OSError from the socket when port binding fails.

`sendOdomReqClient(copterID=1, IP="127.0.0.1", freq=100, port=None)`¶

Function Description: Sends only the Odom request without creating a receiving thread. The return address is taken from RemotSendIP; if not set, 127.0.0.1 is used. The caller must call startCopterSensorReceiver() itself or manage the receiving port.

The parameter meanings, default port, and validation ranges are the same as those of sendOdomReqCopterSim().

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200, port=None)`¶

Function Description: Starts or reuses the shared receiving thread for the specified aircraft, then sends an IMU request with sensorType=0 to CopterSim. The same receiving port can recognize both 40-byte IMU packets and 88-byte Odom packets.

Parameter Name	Type	Default	Description
copterID	`int`	`1`	Target aircraft ID; must be greater than or equal to 1.
IP	`str`	`127.0.0.1`	Address of the host where CopterSim is running.
freq	`int`	`200`	Request frequency; allowed range 1 to 1000 Hz.
port	`int \| None`	`None`	Local return port; defaults to `31000 + copterID - 1`.

Return Value (Returns): None

Exceptions (Raises): Throws ValueError when a parameter is out of range or the return IP is invalid; throws OSError from the socket when port binding fails.

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200, port=None)`¶

Function Description: Sends only the IMU request without creating a receiving thread. The handling rules for port, frequency, and return IP are the same as those of sendImuReqCopterSim().

Return Value (Returns): None

`sendImuReqServe(copterID=1, port=None)`¶

Function Description: An alias for backward-compatible calls; directly forwards to startCopterSensorReceiver(copterID, port); it only starts the receiving end and does not send a request.

Return Value (Returns): None

`startCopterSensorReceiver(copterID=1, port=None)`¶

Function Description: Creates a daemon receiving thread for (copterID, port), or reuses an existing thread with the same key that is still running. The receiving socket binds to 0.0.0.0 with a timeout of 0.2 seconds, and getIMUDataLoop() dispatches both IMU and Odom data.

Parameter Name	Type	Default	Description
copterID	`int`	`1`	ID of the aircraft to which the received data belongs.
port	`int \| None`	`None`	Listening port; must be between 1 and 65535; defaults to being calculated from the aircraft ID.

Return Value (Returns): None; returns directly if the same receiver is already running.

`getIMUDataLoop(copterID, udp_sensor=None, stop_event=None)`¶

Function Description: The internal loop of the IMU/Odom receiving thread. 40-byte packets are decoded according to IMU_DATA_STRUCT, and 88-byte packets with a correct checksum are decoded according to ODOM_DATA_STRUCT; data with an invalid checksum, non-finite values, or a mismatched aircraft ID or coordinate type is discarded.

Parameter Name	Type	Default Value	Description
copterID	`int`	-	The aircraft ID corresponding to this receiving thread.
udp_sensor	`socket \| None`	`None`	Receiving socket; when empty, falls back to using `self.udp_imu`.
stop_event	`threading.Event \| None`	`None`	Stop event for an individual receiving thread.

`getImuData(copterID=1)`¶

Function Description: Thread-safely reads the latest IMU snapshot of the specified aircraft.

Return Value (Returns): Returns a deep copy of imuDataCopterSnapshot if data has been received, otherwise returns None.

`getOdomData(copterID=1)`¶

Function Description: Thread-safely reads the latest Odom snapshot of the specified aircraft. The receiving end rejects duplicate, out-of-order, and rollback frames without a discontinuity flag, and normalizes valid quaternions.

Return Value (Returns): Returns a deep copy of odomDataCopter if data has been received, otherwise returns None.

`get_all_ip()`¶

Function Description: Retrieves all available network interface IP addresses on the local machine Parameters (Args): None Return Value (Returns): A list of all IP addresses on the local machine Exceptions (Raises): None

`isIpLocal(IP)`¶

Function Description: Determines whether a given IP address is a local machine IP address Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
IP	`Any`	Yes	None	The IP address to be checked

Return Value (Returns): bool, returns True if the IP is local, otherwise returns False Exceptions (Raises): None

`StartTimeStmplisten()`¶

Function Description: Starts a listening thread to monitor port 20005 for Rfly time stamps corresponding to various vehicle IDs Parameters (Args): None Return Value (Returns): None Exceptions (Raises): None

`getTimeStmp(CopterID=1)`¶

Function Description: Retrieves the latest Rfly time stamp corresponding to a specified vehicle ID Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
CopterID	`Any`	No	1	ID of the target vehicle

Return Value (Returns): The latest Rfly time stamp for the specified vehicle Exceptions (Raises): None

`endTimeStmplisten()`¶

Function Description: Terminates the time stamp listening thread, stopping the reception of time stamp data Parameters (Args): None Return Value (Returns): None Exceptions (Raises): None

`TimeStmploop()`¶

Function Description: The worker function of the time stamp listening thread, continuously receiving and processing time stamp data sent by CopterSim Parameters (Args): None Return Value (Returns): None Exceptions (Raises): None

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

Function Description: Sends a visual sensor image update request to UE4 Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vs	`Any`	No	`VisionSensorReq()`	Visual sensor request structure
windID	`Any`	No	0	Window ID identifier
IP	`Any`	No	""	IP address of the host running the target UE4; if empty, automatically determined

Return Value (Returns): None Exceptions (Raises): Exception - thrown when the request fails to send

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

Function Description: Updates the visual sensor configuration for a specified RflySim3D window Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
vs	-	No	`VisionSensorReqNew()`	Visual sensor request configuration object
windID	-	No	`0`	RflySim3D window index
IP	-	No	`""`	Target IP address; uses default if empty

Return Value (Returns): No return value Exceptions (Raises): Exception - thrown if an error occurs during sending

`sendUE4Cmd(cmd, windowID=-1)`¶

Function Description: Sends a custom command to UE4 Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
cmd	-	Yes	None	Command content to be sent
windowID	-	No	`-1`	Target RflySim3D window index; -1 indicates sending to all windows

Return Value (Returns): No return value Exceptions (Raises): None

`sendReqToUE4(windID=0, IP="")`¶

Function Description: Sends the visual sensor list to RflySim3D to request corresponding images; the target RflySim3D window index is specified via windID Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
windID	-	No	`0`	Target RflySim3D window index
IP	-	No	`""`	Target IP address; uses default if empty

Return Value (Returns): No return value Exceptions (Raises): None

`setUdpDecodeMode(mode=2)`¶

Function Description: Sets the module-level UDP decoding mode. When mode=1, a single thread sequentially receives packets and decodes; when mode=2, the receiving thread reassembles complete frames and passes them to an independent decoding thread via a bounded queue; this mode is used by default.

Parameter Name	Type	Default Value	Description
mode	`int`	`2`	Only `1` or `2` is allowed.

Return Value (Returns): None

Exceptions (Raises): Throws ValueError when other values are passed in.

`img_udp_thrdNew(udpSok, idx, typeID)`¶

Function Description: Worker function for the UDP image reception thread, handling UDP image reception for a specified visual sensor Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
udpSok	-	Yes	None	UDP socket object
idx	-	Yes	None	Visual sensor index
typeID	-	Yes	None	Sensor type ID

Return Value (Returns): No return value Exceptions (Raises): None

`img_mem_thrd(idxList)`¶

Function Description: Worker function for the shared memory image reception thread, handling image reception for visual sensors in the specified index list Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
idxList	-	Yes	None	List of visual sensor indices to receive

Return Value (Returns): No return value Exceptions (Raises): None

`startImgCap(isRemoteSend=False)`¶

Function Description: Starts the loop to receive images from UE4. When isRemoteSend is True, images obtained from memory are forwarded to the UDP port Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
isRemoteSend	-	No	`False`	Whether to enable remote image forwarding; when enabled, images are forwarded from shared memory to the UDP port

Return Value (Returns): No return value Exceptions (Raises): None

`sendImgUDPNew(idx)`¶

Function Description: Sends the image of the specified visual sensor index via the new UDP protocol Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
idx	-	Yes	None	Visual sensor index

Return Value (Returns): No return value Exceptions (Raises): None

`sendImgBuffer(idx, data)`¶

Function Description: Sends the image buffer data of the specified visual sensor index Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
idx	-	Yes	None	Visual sensor index
data	-	Yes	None	Image data buffer to be sent

Return Value (Returns): No return value Exceptions (Raises): None

`jsonLoad(ChangeMode=-1, jsonPath="")`¶

Function Description: Loads a configuration JSON file and creates a camera list for image acquisition. If ChangeMode>=0, the first item SendProtocol[0] is set to ChangeMode to modify the image transmission mode Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
ChangeMode	-	No	`-1`	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration
jsonPath	-	No	`""`	Path to the configuration JSON file; uses default path if empty

Return Value (Returns): No return value Exceptions (Raises): None

`dictLoad(jsData, ChangeMode=-1)`¶

Function Description: Wrapper for the jsonLoad method, enabling direct reading of sensor data stored in memory as a dictionary. The data structure matches that of the JSON file used in jsonLoad Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
jsData	-	Yes	None	Dictionary of sensor configuration data stored in memory
ChangeMode	-	No	`-1`	Transmission mode override value; if >= 0, overrides the transmission mode in the configuration

Return Value (Returns): No return value Exceptions (Raises): None

`stopRun()`¶

Function Description: Sets the global stop flag, closes all CopterSim IMU/Odom receiving sockets, and waits for the related threads to finish. Parameters (Args): None Return Value (Returns): No return value Exceptions (Raises): None

`stopCopterSensorReceiver(copterID=None, port=None)`¶

Function Description: Stops receivers by aircraft ID, port, or a combination of both; when both parameters are empty, stops all CopterSim IMU/Odom receivers. The method first removes matching entries from the registry, then sets the stop event, closes the socket, and waits for non-current threads for up to 1 second.

Parameter Name	Type	Default Value	Description
copterID	`int \| None`	`None`	Only stop the receiver for this aircraft.
port	`int \| None`	`None`	Only stop the receiver for this port.

Return Value (Returns): None

`SensorPreview(seq_id=-1)`¶

Function Description: Previews images captured by the visual sensor with the specified sequence ID Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
seq_id	-	No	`-1`	Sequence ID of the visual sensor to preview; `-1` indicates previewing all sensors

Return Value (Returns): No return value Exceptions (Raises): None

Advanced Usage Example¶

The following example requests LiDAR point cloud data and continuously updates the new point cloud in vis.Img[0] to an Open3D window.

Reference example: [RflySim installation path]\RflySimAPIs\8.RflySimVision\0.ApiExps\1-UsageAPI\3.PointCloudAPI\2.LidarAPIPointCloudDemo

import sys
import time

import Open3DShow
import VisionCaptureApi


show3d = Open3DShow.Open3DShow()
vis = VisionCaptureApi.VisionCaptureApi()

vis.jsonLoad()
if not vis.sendReqToUE4():
    sys.exit(0)
vis.startImgCap()
show3d.CreatShow(0)

lastTime = time.time()
while True:
    lastTime += 1 / 10.0
    sleepTime = lastTime - time.time()
    if sleepTime > 0:
        time.sleep(sleepTime)
    else:
        lastTime = time.time()

    if vis.hasData[0]:
        show3d.UpdateShow(vis.Img[0])
        vis.hasData[0] = False

Notes and Pitfall Avoidance Guide¶

Duplicate Visual Sensor Addition Issue: When calling addVisSensor multiple times, ensure the input sensor_id is globally unique; duplicate IDs will cause the simulation side to overwrite sensors, resulting in erroneous visual data retrieval.
Asynchronous Data Buffer Overflow: When using a custom queue to cache IMU or visual data, periodically clean up old data; prolonged operation will cause the queue to accumulate and consume excessive memory. This can be mitigated by checking the queue size and popping the oldest data from the front once it exceeds a threshold.
Cross-Host IP Connection Check: Before establishing a connection, call isIpLocal to detect IP properties. If the simulation side runs on a non-local host, ensure the IP port permissions are open; otherwise, the default port may be blocked by the remote firewall, causing request timeouts.
Prerequisites for Time Alignment: Before calling imuDataCopter.AlignTime, time stamp initialization must be completed first. Failure to call RflyTimeStmp.Update to update the reference time stamp will result in alignment deviations, causing SLAM pose jumps.

Changelog¶

2026-09-11: 🐛 fix: Fix the issue of visual frame and odometry time desynchronization [P0]
2026-08-06: Move time base initialization and imgStmp update into the "decode success" branch, executed every frame
2026-03-03: feat: SDK adds IP handling mechanism, compatible with cloud deployment from local version
2026-02-02: Add odom publishing for depth-to-point-cloud conversion
2026-02-02: fix: Optimize protocol
2026-02-01: fix: Comment out path printing
2026-01-31: fix: Update depth-to-point-cloud protocol
2026-01-31: fix: Update communication protocol for depth-to-point-cloud
2026-01-23: Align timestamps for /Depth_Cloud topic
2026-01-22: Merge branch 'master' of http://rflysim.synology.me:8418/FeisiLab/RflySimSDK
2026-01-22: Change IMU coordinate frame orientation for ROS output to ensure consistent FLU frame output; previously, conversion was FRD → FLU, but the platform's IMU angular velocity conforms to FRD, while linear acceleration does not conform to FRD but rather BLD
2025-12-12: fix: Optimize image timestamp synchronization mechanism, resolve image timestamp desynchronization bug on Windows
2025-12-08: Change platform odom data translation in local frame under mode 0 from FRD → FLU
2025-12-07: fix: Add relative position transmission data for LiDAR
2025-12-04: fix: Fix point cloud sensor issue
2025-09-30: Resolve fisheye camera bug under UDP mode
2025-09-29: ROS2 depth-to-point-cloud seq variable has been removed; interface needs adaptation

VisionCaptureApi Interface Documentation¶

Introduction¶

Quick Start¶

Environment and Dependencies¶

Core Interface Description¶

Global Constants and Enumerations¶

Standalone Constants¶

Global/Standalone Functions¶

Queue Class¶

__init__()¶

enqueue(item)¶

dequeue()¶

is_empty()¶

size()¶

RflyTimeStmp Class¶

__init__(iv)¶

Update(iv)¶

VisionSensorReq Class¶

__init__()¶

VisionSensorReqNew Class¶

__init__()¶

imuDataCopter Class¶

__init__(imu_name, node=None)¶

AlignTime(img_time)¶

Imu2ros(node=None)¶

imuDataCopterSnapshot Class¶

__init__()¶

odomDataCopter Class¶

__init__()¶

DistanceSensor Class¶

__init__()¶

OpticalFlowSensor Class¶

__init__()¶

SensorReqCopterSim Class¶

__init__()¶

VisionCaptureApi Class¶

__init__(ip='127.0.0.1')¶

euler2quat(r, p, y, copter, coordinate_frame="global")¶

addVisSensor(vsr=VisionSensorReq())¶

addVisSensor(vsr=VisionSensorReqNew())¶

sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")¶

sendOdomReqCopterSim(copterID=1, IP="127.0.0.1", freq=100, port=None)¶

sendOdomReqClient(copterID=1, IP="127.0.0.1", freq=100, port=None)¶

sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200, port=None)¶

sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200, port=None)¶

sendImuReqServe(copterID=1, port=None)¶

startCopterSensorReceiver(copterID=1, port=None)¶

getIMUDataLoop(copterID, udp_sensor=None, stop_event=None)¶

getImuData(copterID=1)¶

getOdomData(copterID=1)¶

get_all_ip()¶

isIpLocal(IP)¶

StartTimeStmplisten()¶

getTimeStmp(CopterID=1)¶

endTimeStmplisten()¶

TimeStmploop()¶

sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")¶

sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")¶

sendUE4Cmd(cmd, windowID=-1)¶

sendReqToUE4(windID=0, IP="")¶

setUdpDecodeMode(mode=2)¶

img_udp_thrdNew(udpSok, idx, typeID)¶

img_mem_thrd(idxList)¶

startImgCap(isRemoteSend=False)¶

sendImgUDPNew(idx)¶

sendImgBuffer(idx, data)¶

jsonLoad(ChangeMode=-1, jsonPath="")¶

dictLoad(jsData, ChangeMode=-1)¶

stopRun()¶

stopCopterSensorReceiver(copterID=None, port=None)¶

SensorPreview(seq_id=-1)¶

Advanced Usage Example¶

Notes and Pitfall Avoidance Guide¶

Changelog¶

`Queue` Class¶

`init()`¶

`enqueue(item)`¶

`dequeue()`¶

`is_empty()`¶

`size()`¶

`RflyTimeStmp` Class¶

`init(iv)`¶

`Update(iv)`¶

`VisionSensorReq` Class¶

`init()`¶

`VisionSensorReqNew` Class¶

`init()`¶

`imuDataCopter` Class¶

`init(imu_name, node=None)`¶

`AlignTime(img_time)`¶

`Imu2ros(node=None)`¶

`imuDataCopterSnapshot` Class¶

`init()`¶

`odomDataCopter` Class¶

`init()`¶

`DistanceSensor` Class¶

`init()`¶

`OpticalFlowSensor` Class¶

`init()`¶

`SensorReqCopterSim` Class¶

`init()`¶

`VisionCaptureApi` Class¶

`init(ip='127.0.0.1')`¶

`euler2quat(r, p, y, copter, coordinate_frame="global")`¶

`addVisSensor(vsr=VisionSensorReq())`¶

`addVisSensor(vsr=VisionSensorReqNew())`¶

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

`sendOdomReqCopterSim(copterID=1, IP="127.0.0.1", freq=100, port=None)`¶

`sendOdomReqClient(copterID=1, IP="127.0.0.1", freq=100, port=None)`¶

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200, port=None)`¶

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200, port=None)`¶

`sendImuReqServe(copterID=1, port=None)`¶

`startCopterSensorReceiver(copterID=1, port=None)`¶

`getIMUDataLoop(copterID, udp_sensor=None, stop_event=None)`¶

`getImuData(copterID=1)`¶

`getOdomData(copterID=1)`¶

`get_all_ip()`¶

`isIpLocal(IP)`¶

`StartTimeStmplisten()`¶

`getTimeStmp(CopterID=1)`¶

`endTimeStmplisten()`¶

`TimeStmploop()`¶

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

`sendUE4Cmd(cmd, windowID=-1)`¶

`sendReqToUE4(windID=0, IP="")`¶

`setUdpDecodeMode(mode=2)`¶

`img_udp_thrdNew(udpSok, idx, typeID)`¶

`img_mem_thrd(idxList)`¶

`startImgCap(isRemoteSend=False)`¶

`sendImgUDPNew(idx)`¶

`sendImgBuffer(idx, data)`¶

`jsonLoad(ChangeMode=-1, jsonPath="")`¶

`dictLoad(jsData, ChangeMode=-1)`¶

`stopRun()`¶

`stopCopterSensorReceiver(copterID=None, port=None)`¶

`SensorPreview(seq_id=-1)`¶