VisionCaptureApi Interface Documentation¶

Introduction¶

Overview: This document defines the data structures and API classes required by the visual perception module of the RflySim UAV simulation platform, providing foundational interactive support for Python-side requests to UE4 simulation environment for camera images and various sensor data.

In RflySim simulation, visual tasks are central to the development of intelligent algorithms such as UAV autonomous navigation and target detection, requiring parameter exchange and data transmission between the Python and UE4 sides. This module defines structures for sending camera parameter settings and image requests to UE4, as well as encapsulation structures for various perception data—including IMU data and distance sensor data—from CopterSim. It also provides the core VisionCaptureApi class, enabling Python-side acquisition of simulated images from the UE4 simulation environment. It is suitable for developing vision-based UAV simulation algorithms, offering a unified interface specification for visual perception data interaction in simulation environments, supporting monocular and multi-camera simulated data acquisition. Combined with IMU and distance sensor information, it enables multi-sensor fusion simulation tasks.

Quick Start¶

Minimal working example; copy and modify minimal configuration to run.

from RflySimSDK.vision.VisionCaptureApi import VisionCaptureApi, VisionSensorReqNew

# 1. Initialize the visual capture API; connects to local RflySim simulation environment by default
vision_api = VisionCaptureApi(ip="127.0.0.1")

# 2. Create a new visual sensor request object; requests RGB image and pose data by default
sensor_req = VisionSensorReqNew()

# 3. Add the visual sensor request to the capture list
try:
    vision_api.addVisSensor(sensor_req)
except Exception as e:
    print(f"Failed to add visual sensor: {e}")

# 4. Euler angle to quaternion conversion example: roll=0°, pitch=0°, yaw=90°, converted to quaternion
roll = 0
pitch = 0
yaw = 90
quat = vision_api.euler2quat(roll, pitch, yaw, copter="iris", coordinate_frame="global")
print(f"Converted quaternion: {quat}")

Environment and Dependencies¶

Python Environment: >= 3.8.10
Dependencies: Open3DShow, copy, ctrl.IpManager, cv2, email, json, math, mmap, numpy, os, platform, psutil, re, socket, struct, subprocess, sys, threading, time, warnings
Prerequisites: Before calling this interface, ensure that the RflySimSDK vision module has been correctly imported.

Core Interface Description¶

The module VisionCaptureApi.py includes configuration variables, helper functions, and the core business class.

Global Constants and Enumerations¶

This section lists all globally accessible constants and enumeration definitions directly referenceable within the module.

Standalone Constants¶

None

Global/Standalone Functions¶

None

`Queue` Class¶

Provides a basic queue data structure for storing and managing elements in a first-in-first-out (FIFO) manner.

`init()`¶

Function Description: Initializes an empty queue instance
Parameters (Args):
None
Returns:

Queue instance object

Exceptions (Raises):
None

`enqueue(item)`¶

Function Description: Adds an element to the tail of the queue
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`item`	`Any type`	Yes	-	Element to be added to the queue

Returns:

None

Exceptions (Raises):
None

`dequeue()`¶

Function Description: Removes and returns the element at the head of the queue
Parameters (Args):
None
Returns:

Element at the head of the queue

Exceptions (Raises):
None

`is_empty()`¶

Function Description: Determines whether the current queue is empty
Parameters (Args):
None
Returns:

bool: Returns True if the queue is empty, otherwise returns False

Exceptions (Raises):
None

`size()`¶

Function Description: Retrieves the current number of elements stored in the queue
Parameters (Args):
None
Returns:

int: Number of elements in the queue

Exceptions (Raises):
None

Example:

from RflySimSDK.vision import Queue

# Create a queue and add elements
q = Queue()
q.enqueue("first frame")
q.enqueue("second frame")

# Check queue size
print(q.size())  # Outputs: 2

# Dequeue an element
item = q.dequeue()
print(item)  # Outputs: first frame

# Check if queue is empty
print(q.is_empty())  # Outputs: False

---

### `RflyTimeStmp` Class

A utility class for managing timestamp data of drone vision data, capable of storing and updating timestamp values, used in conjunction with the vision capture module.

#### `__init__(iv)`

**Function Description**: Initializes an `RflyTimeStmp` instance and sets the initial timestamp value.  
**Parameters (Args)**:

| Parameter Name | Type | Required | Default | Description |
| :--- | :--- | :---: | :--- | :--- |
| `iv` | Any numeric type | Yes | - | Initial timestamp value |

**Return Value (Returns)**:

- `RflyTimeStmp` instance object

**Exceptions (Raises)**:
- None

---

#### `Update(iv)`

**Function Description**: Updates the stored timestamp value to the new input value.  
**Parameters (Args)**:

| Parameter Name | Type | Required | Default | Description |
| :--- | :--- | :---: | :--- | :--- |
| `iv` | Any numeric type | Yes | - | New timestamp value to be updated |

**Return Value (Returns)**:
- None  
**Exceptions (Raises)**:
- None

**Example**:

```python
from RflySimSDK.vision import RflyTimeStmp
# Initialize timestamp to 0
ts = RflyTimeStmp(0)
# Update timestamp to current simulation time 12.34
ts.Update(12.34)

`VisionSensorReq` Class¶

A C++ structure corresponding to a Python class used to request and configure camera parameters in the UE4 simulation environment; applicable for configuring vision sensor parameters in RflySim drone vision simulation.

`init()`¶

Function Description: Initializes a VisionSensorReq instance; all parameter members are created according to the default C++ structure definition.
Parameters (Args): - None
Return Value (Returns):

VisionSensorReq instance object

Exceptions (Raises): - None

Example:

from RflySimSDK.vision import VisionSensorReq
# Create a vision sensor parameter request object
sensor_req = VisionSensorReq()
# Configure the target aircraft ID
sensor_req.TargetCopter = 1
# Set the camera mounting position
sensor_req.SensorPosXYZ = [0, 0, 0.1]

`VisionSensorReqNew` Class¶

A C++-style structure used to send requests and configure vision sensor parameters to the UE4 simulation environment; used in RflySim drone vision simulation to define vision sensor configuration information.

`init()`¶

Function Description: Initializes a VisionSensorReqNew instance object.
Parameters (Args): - None
Return Value (Returns):

VisionSensorReqNew instance object

Exceptions (Raises): - None

`imuDataCopter` Class¶

A data structure used to receive IMU sensor data from CopterSim, storing checksum, message sequence number, timestamp, accelerometer data, and gyroscope data.

`init(imu_name, node=None)`¶

Function Description: Initializes an imuDataCopter instance, setting the IMU data ROS topic name and ROS node.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`imu_name`	`str`	No	`"/rflysim/imu"`	ROS topic name for IMU data
`node`	`Any`	No	`None`	ROS node object used to publish IMU data

Return Value (Returns):

imuDataCopter instance object

Exceptions (Raises):

None

`AlignTime(img_time)`¶

Function Description: Aligns the IMU data timestamp with the image timestamp to match IMU data at the corresponding moment.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`img_time`	`double`	Yes	-	Image timestamp to be aligned

Return Value (Returns):

Aligned and matched IMU data

Exceptions (Raises):

None

`Imu2ros(node=None)`¶

Function Description: Converts the current IMU data into ROS format and publishes it.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`node`	`Any`	No	`None`	ROS node object used to publish IMU data; if `None`, uses the node set during initialization

Return Value (Returns):

None

Exceptions (Raises):

None

Example:

from RflySimSDK.vision import imuDataCopter
# Initialize IMU data receiver object
imu = imuDataCopter()
# Align with image timestamp 123.45 to obtain corresponding IMU data
aligned_imu = imu.AlignTime(123.45)

`DistanceSensor` Class¶

A distance sensor functional class used to implement distance sensor-related simulation logic in the RflySim drone simulation.

`init()`¶

Function Description: Initializes an instance of the DistanceSensor class.
Arguments (Args):
None
Returns:
- An instance of the DistanceSensor class

Exceptions (Raises):
None

`SensorReqCopterSim` Class¶

A structure class used to send sensor data requests to UE4, corresponding to a C++ structure definition. It is primarily used in the RflySim drone simulation to transmit sensor request configuration information.

`init()`¶

Function Description: Initializes an empty instance of the sensor request structure.
Arguments (Args):
None
Returns:
- An instance of the SensorReqCopterSim class

Exceptions (Raises):
None

`VisionCaptureApi` Class¶

This is the Python-side API class for retrieving images from UE4.

`init(ip='127.0.0.1')`¶

Function Description: Initializes an instance of the visual image capture API and establishes a connection to the UE4 image service.
Arguments (Args):

Parameter Name	Type	Default Value	Description
ip	`str`	`127.0.0.1`	IP address where the UE4 image service is running; use the default localhost loopback address when running locally

Returns: None
Exceptions (Raises): None

`euler2quat(r, p, y, copter, coordinate_frame="global")`¶

Function Description: Converts Euler angles to quaternions, supporting conversions for different vehicle types and coordinate frames.
Arguments (Args):

Parameter Name	Type	Required	Default Value	Description
r	`Any`	Yes	None	Roll angle Euler component
p	`Any`	Yes	None	Pitch angle Euler component
y	`Any`	Yes	None	Yaw angle Euler component
copter	`str`	Yes	None	Vehicle type identifier
coordinate_frame	`Any`	No	`"global"`	Coordinate frame type; defaults to global coordinate frame

Returns: The converted quaternion
Exceptions (Raises): None

`addVisSensor(vsr=VisionSensorReq())`¶

Function Description: Adds a new legacy VisionSensorReq visual sensor request structure to the visual sensor list.
Arguments (Args):

Parameter Name	Type	Required	Default Value	Description
vsr	`Any`	No	`VisionSensorReq()`	Legacy visual sensor request structure to be added

Returns: None
Exceptions (Raises): Exception - Thrown if addition fails

`addVisSensor(vsr=VisionSensorReqNew())`¶

Function Description: Adds a new updated VisionSensorReqNew visual sensor request structure to the visual sensor list.
Arguments (Args):

Parameter Name	Type	Required	Default Value	Description
vsr	`Any`	No	`VisionSensorReqNew()`	Updated visual sensor request structure to be added

Returns: None
Exceptions (Raises): Exception - Thrown if addition fails

`sendReqToCopterSim(srcs=SensorReqCopterSim(), copterID=1, IP="127.0.0.1")`¶

Function Description: Sends a SensorReqCopterSim-type UDP request message to CopterSim to request sensor data. The target CopterSim instance is specified via copterID.
Arguments (Args):

Parameter Name	Type	Required	Default Value	Description
srcs	`Any`	No	`SensorReqCopterSim()`	Sensor request structure
copterID	`Any`	No	1	ID of the target vehicle; specifies the index of the CopterSim instance to request data from
IP	`Any`	No	`"127.0.0.1"`	IP address of the host where the target CopterSim is running

Returns: None
Exceptions (Raises): None

`sendImuReqCopterSim(copterID=1, IP="127.0.0.1", freq=200)`¶

Function Description: Sends an IMU data request command to CopterSim. This function initializes a thread to listen for incoming IMU data.
The actual port = base port + vehicle ID - 1.
Arguments (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	No	1	ID of the target vehicle
IP	`Any`	No	`"127.0.0.1"`	IP address of the target host to which the request is sent
freq	`Any`	No	200	Requested IMU data transmission frequency

Returns: None
Exceptions (Raises): None

`sendImuReqClient(copterID=1, IP="127.0.0.1", freq=200)`¶

Function Description: Sends an IMU data request command to CopterSim. The actual port = base port + vehicle ID - 1.
Arguments (Args):

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	No	1	ID of the target vehicle
IP	`Any`	No	"127.0.0.1"	IP address of the target host to which the request is sent
freq	`Any`	No	200	Requested IMU data transmission frequency

Returns: None
Raises: None

`sendImuReqServe(copterID=1)`¶

Function Description: Sends an IMU data request command to CopterSim. This function initializes a thread to listen for and receive IMU data, where the actual port = base port + vehicle ID - 1.
Arguments:

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	No	1	ID of the target vehicle

Returns: None
Raises: None

`getIMUDataLoop(copterID)`¶

Function Description: Cyclically retrieves IMU data for a specified vehicle, serving as the worker function for the IMU listening thread.
Arguments:

Parameter Name	Type	Required	Default Value	Description
copterID	`Any`	Yes	None	ID of the vehicle whose data is to be retrieved

Returns: None
Raises: None

`get_all_ip()`¶

Function Description: Retrieves all available network interface IP addresses on the local machine.
Arguments: None
Returns: A list of all IP addresses on the local machine
Raises: None

`isIpLocal(IP)`¶

Function Description: Determines whether a given IP address is a local machine IP address.
Arguments:

Parameter Name	Type	Required	Default Value	Description
IP	`Any`	Yes	None	The IP address to be checked

Returns: bool — returns True if the IP is local, otherwise False
Raises: None

`StartTimeStmplisten()`¶

Function Description: Starts a listening thread to monitor port 20005 for Rfly time stamps corresponding to various vehicle IDs.
Arguments: None
Returns: None
Raises: None

`getTimeStmp(CopterID=1)`¶

Function Description: Retrieves the latest Rfly time stamp corresponding to a specified vehicle ID.
Arguments:

Parameter Name	Type	Required	Default Value	Description
CopterID	`Any`	No	1	ID of the target vehicle

Returns: The latest Rfly time stamp for the specified vehicle
Raises: None

`endTimeStmplisten()`¶

Function Description: Terminates the time stamp listening thread, stopping the reception of time stamp data.
Arguments: None
Returns: None
Raises: None

`TimeStmploop()`¶

Function Description: The worker function of the time stamp listening thread, continuously receiving and processing time stamp data sent by CopterSim.
Arguments: None
Returns: None
Raises: None

`sendUpdateUEImage(vs=VisionSensorReq(), windID=0, IP="")`¶

Function Description: Sends a visual sensor image update request to UE4.
Arguments:

Parameter Name	Type	Required	Default Value	Description
vs	`Any`	No	`VisionSensorReq()`	Visual sensor request structure
windID	`Any`	No	0	Window ID identifier
IP	`Any`	No	""	IP address of the host running the target UE4; if empty, automatically determined

Returns: None
Raises: Exception — thrown when the request fails to send

`sendUpdateUEImaged(vs=VisionSensorReqNew(), windID=0, IP="")`¶

Function Description: Updates the visual sensor configuration for a specified RflySim3D window.
Arguments:

Parameter Name	Type	Required	Default Value	Description
vs	-	No	`VisionSensorReqNew()`	Visual sensor request configuration object
windID	-	No	`0`	RflySim3D window index
IP	-	No	`""`	Target IP address; uses default if empty

Returns: None
Raises: Exception — thrown when an error occurs during sending

`sendUE4Cmd(cmd, windowID=-1)`¶

Function Description: Sends a custom command to UE4.
Arguments:

Parameter Name	Type	Required	Default Value	Description
cmd	-	Yes	None	Command content to be sent
windowID	-	No	`-1`	Target RflySim3D window index; `-1` indicates sending to all windows

Returns: None
Raises: None

`sendReqToUE4(windID=0, IP="")`¶

Function Description: Sends a list of visual sensors to RflySim3D to request corresponding images; the target RflySim3D window is specified by windID.
Arguments:

Parameter Name	Type	Required	Default Value	Description
windID	-	No	`0`	Target window index for RflySim3D
IP	-	No	`""`	Target IP address; uses default address if empty