RflySim Vision and Sensor Communication Protocol

This document describes the vision sensor data transmission, ROS/ROS 2 forwarding, CopterSim IMU/Odom interface, and gimbal communication protocol currently implemented in VisionCaptureApi.py.

Unless otherwise specified:

• Multi-byte values are interpreted in the little-endian format of the current RflySim runtime environment.
• SeqID is the sensor ID; sensor{SeqID} in ROS topics uses this value.
• TypeID is the sensor type and is not equivalent to SeqID.
• ROS forwarding is controlled by the module-level variable isEnableRosTrans in VisionCaptureApi.py, which must be enabled before creating a VisionCaptureApi instance. For example, first import VisionCaptureApi as vis_api, then set vis_api.isEnableRosTrans = True.
• UE4 and UE5 use the same Python receiving interface. UE5 uses the GPU to generate some point clouds, but external transmission still uses the int16 point coordinate encoding described in this document.

1. Data Transmission

1.1 SendProtocol

Both VisionSensorReq.SendProtocol and VisionSensorReqNew.SendProtocol are arrays composed of 8 uint16 values.

Index	Meaning
[0]	Transmission mode
[1:5]	Four bytes of the target IPv4 address
[5]	Target UDP port
[6]	UDP fragment payload size, usually no more than 60000 bytes
[7]	Sensor-specific function bits; for point clouds, used to select the three-channel or four-channel wire format

The values of SendProtocol[0] are as follows.

Value	Transmission Method	Description
0	Shared memory	Raw data, low-latency local transmission; jsonLoad() switches to UDP by default on Linux
1	UDP	Images usually use encoded data such as JPEG; point clouds and structured sensors use custom binary payloads
2	UDP raw image	Image pixels are transmitted uncompressed; only applicable to image-type sensors
3	UDP PNG	Images use lossless PNG compression; only applicable to image-type sensors

TypeID 5, 7, 10, 20–23, 30, and 31 use custom binary payloads and should not be decoded by invoking video stream modes as ordinary images.

1.2 UDP Fragment Packet Header

A UDP data frame may be split into multiple packets. The current receiver identifies two types of packet headers based on the checksum.

24-Byte Packet Header

The checksum is 1234567890, and the Python format is 4i1d.

Offset	Field	Type	Description
0	checksum	int32	Fixed at 1234567890
4	packet_len	int32	Total length of the current UDP packet, including the packet header
8	packet_seq	int32	Current fragment sequence number, starting from 0
12	packet_count	int32	Total number of fragments in the current data frame
16	timestamp	float64	Simulation time when RflySim3D/UE generated the frame

32-Byte Packet Header

The checksum is 1234567893, and the Python format is 6i1d.

Offset	Field	Type	Description
0	checksum	int32	Fixed at 1234567893
4	packet_len	int32	Total length of the current UDP packet, including the packet header
8	packet_seq	int32	Current fragment sequence number, starting from 0
12	packet_count	int32	Total number of fragments in the current data frame
16	frame_id	int32	Data frame ID
20	reserved	int32	Reserved field
24	timestamp	float64	Simulation time when RflySim3D/UE generated the frame

The receiver uses the timestamp to distinguish concurrent data frames and concatenates payloads according to packet_seq. When implementing a custom receiver, both packet header types should be supported.

Byte Order

The current Python code uses the native struct format for vision UDP packet headers. On the x86/x64 little-endian environments supported by the platform, the above formats are 24 and 32 bytes, respectively. Cross-architecture implementations should explicitly fix little-endian and field widths, and should not rely on compiler struct alignment.

1.3 Shared Memory Header

Shared memory data does not use the UDP fragment packet header. Each shared memory region begins with 9 bytes of control information:

Offset	Field	Type	Description
0	state	uint8	Write, read, and completion states
1	timestamp	float64	Simulation timestamp
9	payload	bytes	Sensor payload

The shared memory name is RflySim3DImg_{SeqID}. Images are raw pixels; point clouds and structured sensors continue to use the payload format corresponding to each TypeID.

2. Sensor Data

2.1 TypeID Index

TypeID	Sensor	Python Output
1	RGB camera	H×W×3 uint8, BGR
2	Depth camera	H×W uint16 or H×W×1 uint16
3	Grayscale camera	H×W uint8 or H×W×1 uint8
4	Semantic segmentation camera	H×W×3 uint8, BGR
5	Laser rangefinder	DistanceSensor
7	Depth-to-point-cloud	Point cloud array
8	Fisheye camera	H×W×3 uint8, BGR
9	Gimbal camera	H×W×3 uint8, BGR
10	Optical flow sensor	OpticalFlowSensor
20	Vehicle/sensor frame LiDAR	Point cloud array
21	World frame LiDAR	Point cloud array
22	Livox petal-scan LiDAR	Point cloud array
23	Livox Mid-360 LiDAR	Point cloud array
30	Simple visual object detection	List of target boxes
31	Simple radar object detection	List of target relative positions
40	Infrared grayscale camera	H×W uint8
41	Infrared color camera	H×W×3 uint8, BGR

2.2 Images and ROS Topics

When ROS forwarding is enabled, image-type sensors publish sensor_msgs/Image:

TypeID	ROS Encoding	Topic
1	bgr8	/rflysim/sensor{SeqID}/img_rgb
2	mono16	/rflysim/sensor{SeqID}/img_depth
3	mono8	/rflysim/sensor{SeqID}/img_gray
4	bgr8	/rflysim/sensor{SeqID}/img_Segmentation
8	bgr8	/rflysim/sensor{SeqID}/fisheye
9	bgr8	/rflysim/sensor{SeqID}/img_cine
40	mono8	/rflysim/sensor{SeqID}/img_Infrared_Gray
41	bgr8	/rflysim/sensor{SeqID}/img_Infrared

ROS topics are case-sensitive. The message frame_id defaults to map, and can also be configured via sensors_frame_id in tf_cfg.yaml.

2.3 Point Cloud Common Payload

The UDP and shared memory payloads of TypeID 7, 20, 21, 22, and 23 use the same point cloud body. The following offsets are calculated from the start of the sensor payload and do not include the UDP packet header or the 9-byte control header of shared memory.

Offset	Field	Type	Description
0	copter_id	int32	Vehicle ID to which the sensor is bound
4	axis_type	int32	Pose reference mode
8	position[3]	float32[3]	Position
20	euler[3]	float32[3]	Roll, Pitch, Yaw
32	point_count	int32	Number of points
36	points	int16[]	Point cloud data

axis_type corresponds to the sensor's AxisMask configuration:

Value	Meaning
0	Use the absolute pose of the bound vehicle
1	Use the absolute pose of the sensor
2	Use the pose of the sensor relative to its initial position

The point data format is determined by SendProtocol[7]:

SendProtocol[7]	Wire Format	Bytes per Point
0	int16 [x,y,z]	6
Greater than 0	int16 [x,y,z,stencil]	8

The decoding formula for XYZ is:

xyz_m = xyz_int16 * otherParams[0] / 32767

The fourth channel is the semantic/Stencil encoding and does not participate in distance scaling. To keep the downstream array shape stable, the current Python interface converts both wire formats to N×4; the fourth column of three-channel payloads is padded with 0. The UDP path outputs float32, while the shared memory path currently outputs float64.

2.4 LiDAR ROS Data

TypeID 20–23 publish sensor_msgs/PointCloud2:

TypeID	Topic
20	/rflysim/sensor{SeqID}/vehicle_lidar
21	/rflysim/sensor{SeqID}/global_lidar
22	/rflysim/sensor{SeqID}/livox_lidar
23	/rflysim/sensor{SeqID}/mid360_lidar

The current UDP publishing path uses four FLOAT32 fields x/y/z/seg, with offsets 0, 4, 8, and 12, respectively, and point_step=16. The fourth field stores the Stencil value above; the fourth field corresponding to a three-channel payload is 0.

2.5 Depth-to-Point-Cloud

TypeID 7 uses the Point Cloud Common Payload, rather than per-point float32 [x,y,z].

The ROS topic is:

/rflysim/sensor{SeqID}/Depth_Cloud

The message type is sensor_msgs/PointCloud2. The layout declaration of the current ROS message is inconsistent with the actual data length; see Current Implementation Limitations for details.

2.6 Laser Rangefinder

TypeID 5 can be used independently or as part of the gimbal rangefinding function. The standalone rangefinder sensor payload is 56 bytes:

Offset	Field	Type	Description
0	distance	float32	Distance from the ray origin to the hit point
4	copter_id	int32	Bound vehicle ID
8	ray_start[3]	float32[3]	Ray origin
20	angle_euler[3]	float32[3]	Sensor Euler angles
32	impact_point[3]	float32[3]	Hit point
44	box_origin[3]	float32[3]	Center of the bounding box of the hit object

VisionCaptureApi saves the result as a DistanceSensor object in self.Img[idx]. The current version does not create a /rflysim/sensor{SeqID}/distance ROS publisher for TypeID 5.

2.7 Optical Flow

The UDP payload of TypeID 10 is 34 bytes:

Offset	Field	Type
0	time_usec	uint64
8	sensor_id	uint8
9	flow_x	int16
11	flow_y	int16
13	flow_comp_m_x	float32
17	flow_comp_m_y	float32
21	quality	uint8
22	ground_distance	float32
26	flow_rate_x	float32
30	flow_rate_y	float32

The received result is stored in VisionCaptureApi.OpticalFlowSensor. The current SDK does not create a ROS publisher for this data, and the shared memory receive function has no dedicated branch for TypeID 10.

2.8 Simple Target Sensor

TypeID 30 outputs 2D detection boxes for targets within the field of view:

int32 target_count
repeat target_count times:
    int32   copter_id
    float32 credibility
    float32 min_x
    float32 min_y
    float32 max_x
    float32 max_y

TypeID 31 outputs the relative positions of omnidirectionally detected targets:

int32 target_count
repeat target_count times:
    int32   copter_id
    float32 relative_x
    float32 relative_y
    float32 relative_z

otherParams[0] is the maximum detection distance, and otherParams[1] is the maximum number of targets. Both types of results are obtained via self.Img[idx], and there is currently no built-in ROS message publisher.

3. CopterSim IMU and Odometry

3.1 Request Protocol

VisionCaptureApi requests data from CopterSim via SensorReqCopterSim. The request structure is 36 bytes, little-endian <4H4B6f:

Field	Type	Description
checksum	uint16	Request checksum
sensor_type	uint16	0=IMU, 1=Odom
update_freq	uint16	Update frequency; the current interface is limited to 1–1000 Hz
port	uint16	Return UDP port
ip[4]	uint8[4]	Return IPv4 address
params[6]	float32[6]	Reserved parameters

The request is sent to 30100 + (copterID - 1) * 2. The default return port is 31000 + copterID - 1; IMU and Odom can reuse the same receive thread.

3.2 IMU

Call sendImuReqCopterSim() to request IMU. The returned message is 40 bytes, little-endian <iid6f:

Field	Type	Description
checksum	int32	Fixed at 1234567898
sequence	int32	Message sequence number
simulation_time	float64	Simulation time
acceleration[3]	float32[3]	Raw acceleration
angular_rate[3]	float32[3]	Raw angular velocity

The current ROS mapping is:

linear_acceleration = [-acc[0],  acc[1], -acc[2]]
angular_velocity    = [ rate[0], -rate[1], -rate[2]]

By default, sensor_msgs/Imu is published to /rflysim/imu; this can be modified via imu_topic_name and imu_frame_id in tf_cfg.yaml. The interface does not provide attitude estimation; the attitude quaternion in the message is 0, and the first element of the attitude covariance is set to -1 to indicate unknown.

3.3 Standalone Odom

Call sendOdomReqCopterSim() to request standalone Odom. Alternatively, sendOdomReqClient() can be used to send only the request and manage the receiving end yourself. The returned message is 88 bytes, little-endian <IIdIHH3d4f3f3f:

Field	Type	Description
checksum	uint32	Fixed at 1234567888
sequence	uint32	Message sequence number
simulation_time	float64	Simulation time
copter_id	uint32	Vehicle ID
frame_type	uint16	Currently valid value is 1, indicating world NED, body FRD
flags	uint16	Data validity bits and discontinuity flag
position[3]	float64[3]	NED world position
quaternion_wxyz[4]	float32[4]	Attitude quaternion from NED to FRD
linear_velocity[3]	float32[3]	NED linear velocity
angular_velocity[3]	float32[3]	FRD angular velocity

flags is defined as follows:

Bit	Meaning
1 << 0	Position valid
1 << 1	Attitude valid
1 << 2	Linear velocity valid
1 << 3	Angular velocity valid
1 << 5	Data discontinuous; local coordinate origin needs to be re-established

Data is cached in odomDataByCopter by copter_id and retrieved via getOdomData(copterID).

After ROS forwarding is enabled, the interface converts NED/FRD to the NWU/FLU used by ROS and publishes:

Topic	header.frame_id	child_frame_id
/rflysim/uav{copterID}/global/odom	map	base_link{copterID}
/rflysim/uav{copterID}/local/odom	odom{copterID}	base_link{copterID}

The local Odom uses the first valid sample as its origin. After receiving the discontinuity flag, the interface clears that aircraft's initial transform and re-establishes the local coordinate frame on the next frame.

3.4 Pose Carried by Point Cloud

The point cloud payloads of TypeID 7 and 20–23 still carry copter_id, axis_type, and position/euler, so that the point cloud and pose share the same timestamp. This compatibility path mainly provides pose; velocity fields remain at their default values.

When standalone Odom has already been received for the same copter_id, the SDK uses the standalone Odom publisher and no longer publishes the Odom carried by the point cloud.

4. Gimbal UDP Protocol

4.1 Control Request

The gimbal uses TypeID 9 and VisionSensorReqNew. The structure is 148 bytes, with Python format 2H1I14H28f:

struct VisionSensorReqNew {
    uint16 checksum;             // Control request is 12345
    uint16 seq_id;
    uint32 bitmask;
    uint16 type_id;
    uint16 target_copter;
    uint16 target_mount_type;
    uint16 data_width;
    uint16 data_height;
    uint16 data_check_freq;
    uint16 send_protocol[8];
    float camera_fov;
    float sensor_pos_xyz[3];
    float eular_or_quat;
    float sensor_ang_eular[3];
    float sensor_ang_quat[4];
    float other_params[16];
};

The control request is sent via UDP to 20010 + windID. The Python call interface is:

vis.sendUpdateUEImaged(vs, windID=0, IP="")

Note that the function name is sendUpdateUEImaged. sendUpdateUEImage without the trailing d only accepts the older VisionSensorReq.

4.2 bitmask

Multiple control items can be combined with bitwise OR:

bit	Valid Field	Description
1 << 1	CameraFOV	Set field of view
1 << 2	SensorAngEular/Quat	Set target attitude
1 << 3	EularOrQuat, SensorAngQuat	Use quaternion attitude mode
1 << 4	otherParams[0]	Set focal length, unit: mm
1 << 5	otherParams[1]	Center when greater than 0
1 << 6	otherParams[2:4]	Pitch, Yaw angular velocity, unit: deg/s
1 << 7	otherParams[4]	Optical zoom
1 << 8	otherParams[5]	Track target by CopterID or specified position
1 << 9	otherParams[6]	Toggle gimbal laser rangefinder
1 << 10	otherParams[7]	0=RGB, 1=Infrared color, 2=Infrared grayscale
1 << 11	otherParams[8:11]	AI box-select all targets or pixel-point selection

In AI mode, otherParams[8]=1 means box-select targets within the field of view; otherParams[8]=2 means use the pixel coordinates in otherParams[9], [10] to select a target.

4.3 Status Reception

UE4CtrlAPI receives gimbal status on UDP port 20006. The returned data also uses 2H1I14H28f, with a length of 148 bytes, and the return checksum is 12346. The data is stored in:

ue.CamDataVect1

Commonly used fields include CameraFOV, SensorAngEular, SensorAngQuat, and otherParams. The current gimbal example reads the ranging result from otherParams[7]. Different RflySim3D/UE5 versions may extend otherParams; before use, the corresponding version's example should also be checked.

5. RosTrans Gimbal Topics

RosTrans currently uses rflysim_msgs/GimbalCtrl and rflysim_msgs/GimbalStatus, and no longer uses the CameraCtrl/CameraAICtrl/CaramerStatus/CameraParams messages from the old documentation.

5.1 Gimbal Control

Item	Value
Message Type	rflysim_msgs/GimbalCtrl
Topic	/onboard/gimbal/common/control
Routing Method	The receiver filters by target_id

Main values of ctrl_type:

Value	Function
4	Angular velocity control or centering
8	Optical zoom
120	Continuous laser toggle
121	Single laser ranging
122	Pixel coordinate tracking
123	Stop tracking or exit GPS stare
124	GPS coordinate stare
125	Night mode

The message also provides corresponding fields such as yaw_speed_dps, pitch_speed_dps, pixel_x/y, laser_on, GPS stare coordinates, and target_id.

5.2 Gimbal Status

Item	Value
Message Type	rflysim_msgs/GimbalStatus
Topic	/onboard/gimbal/state/status
Routing Field	source_id

The status message includes:

• Camera intrinsics fx/fy/cx/cy and distortion parameters;
• FRD mounting extrinsics from the aircraft to the gimbal base;
• Attitude and validity bits from gimbal to camera and from NED to camera;
• laser_range_valid and laser_range_m.

The direct UDP gimbal interface and the RosTrans ROS interface are two separate entry points. The former uses VisionSensorReqNew and ports 20010/20006, while the latter uses the aforementioned ROS custom messages. The message structures should not be mixed.

6. Current Implementation Limitations

The following items are actual limitations of the current VisionCaptureApi.py. Special attention is required when developing custom receivers or ROS nodes:

1. The Python point cloud array for TypeID 7 is fixed at four columns. Both the UDP and shared memory ROS paths only declare three fields x/y/z and set point_step=12, but what is written is still four columns of float32 data, causing the actual length of PointCloud2.data to be inconsistent with the message layout. This version should not use the built-in TypeID 7 ROS publishing as a stable protocol.
2. The fourth ROS field for TypeID 20–23 is named seg in the UDP path and w in the shared memory path. The field contents in both cases come from the unscaled Stencil channel, and subscribers should not interpret w as a quaternion component.
3. The UDP ROS path for TypeID 8 supports /fisheye, but the shared memory ROS message construction branch currently omits TypeID 8.
4. TypeIDs 5, 10, 30, and 31 can be decoded by Python, but there is currently no built-in ROS publishing branch. When global ROS forwarding is enabled, it is recommended to publish messages for these types yourself, or avoid letting them enter the SDK's generic ROS publishing path.
5. The visual UDP packet header currently uses host byte order; the standalone IMU/Odom already uses explicit little-endian. Custom cross-platform implementations should fix the format to little-endian.

7. Related Documents

• VisionCaptureApi Python Interface
• UE4CtrlAPI Python Interface
• ROS and MAVROS Interface
• RflyRosStart Interface

MAVROS process startup, PX4 links, and multi-vehicle namespaces belong to flight control and ROS communication, not the vision sensor wire protocol, so they are described uniformly in the ROS and RflyRosStart documents.