8.3 Frequently Asked Questions
8.3.1 What is the intrinsic parameter matrix of the depth camera?
Answer:
The intrinsic parameter matrix of a depth camera describes its internal parameters, including focal length, principal point position, and distortion. It is typically a 3×3 matrix:
K = [[f_x, 0, c_x], [0, f_y, c_y], [0, 0, 1]]
Where f_x and f_y are the focal lengths in the horizontal and vertical directions, and c_x and c_y are the principal point (optical center) coordinates. The specific values depend on the camera model — consult your camera's technical documentation. Some depth cameras also include radial and tangential distortion parameters.
8.3.2 How can a developed face recognition or altitude control algorithm be ported to or verified in the simulation environment?
Answer:
For face recognition: Write the algorithm in Python and run it on the platform. Replace the image source with the corresponding camera from the platform. Alternatively, use the NX hardware-in-the-loop mode — develop in C++/ROS, perform HIL visual simulation, then connect the NX to the real aircraft and switch the image source to a physical camera for real flight.
For altitude control: Place the algorithm directly in Simulink and flash it to the flight controller via the low-level flight control development process (Chapter 5).
8.3.3 During software simulation, how do I apply my own Python algorithm? Is there API documentation and demo code?
Answer:
Refer to Chapter 8 PPT Section 2.1:
[Installation Directory]\RflySimAPIs\8.RflySimVision\PPT.pdf
8.3.4 How to modify the UAV flight path in the routine to match trajectory points generated by a target search algorithm?
Answer:
Using the RflySim Python interface: Send trajectory points using
PX4MavCtrl.py. UsegetPos()to get the aircraft's current position and implement a waypoint-reached judgment mechanism. Related functions:SendPosNED(),SendPosFRD(),SendPosXXX()— note the coordinate system differences and perform necessary conversions.Using mavros: Publish to
/mavros/setpoint_position/localor/mavros/setpoint_raw/local. Obtain position via/mavros/local_position/pose. The same waypoint judgment mechanism and coordinate system conversions are required.
8.3.5 Can the depth camera provide left/right/up/down distance information in addition to depth?
Answer:
Yes. Depth cameras provide 3D coordinates of each pixel in the camera coordinate system. By converting depth values (parallax values) to actual distances using the intrinsic parameter matrix and triangulation, you can obtain left/right/up/down distance information. This process is called depth resolution and is used for object distance estimation, pose estimation, 3D reconstruction, etc.
8.3.6 How to get the true pose in RflySim, and how to enable visual positioning?
Answer:
The RflySim platform transmits data via MAVLink. In ROS, subscribe to the mavros pose topic; in Python, use the PX4MavCtrlV4.py interface. To use visual positioning, set the EKF2_AID_MASK parameter in QGC to use vision as the data source, and set EKF2_HGT_MODE to vision. Visual position/pose data can be sent via mavros.
8.3.7 Can the JSON configuration file set pixel size in addition to image resolution? What is the default pixel size?
Answer:
The RflySim platform's config does not currently support pixel size customization.
8.3.8 How to modify RflySim's LiDAR field of view? Can it be changed to 64-line?
Answer:
Yes, it can be changed to 64-line. Adjust the number of scan lines via DataHeight, and configure resolution via DataWidth and the field-of-view angle. For LiDAR parameter details and code parsing, refer to:
[Installation Directory]\RflySimAPIs\8.RflySimVision\PPT.pdf
8.3.9 How to obtain images captured by PX4's simulated camera in Gazebo at the QGC ground station?
Answer:
This requires additional tools:
- Set up the camera model in Gazebo: Ensure the camera model is properly configured in the simulation environment.
- Start the PX4 simulation: Launch PX4 (jMAVSim or Gazebo) and connect to QGC.
- View Gazebo image topics: While running, access camera images via the Gazebo image topics (topic name depends on your camera model configuration).
- Use ROS tools: If using ROS, use tools like
rqt_image_viewto view image topics. - Use ROS Bridge: If QGC supports ROS Bridge, forward ROS image topics to QGC for viewing.
Exact steps may vary with PX4, Gazebo, and QGC versions. Consult the relevant documentation.
8.3.10 For the ROS control examples in the vision module, can ROS also be used to control the real aircraft (e.g., path planning)?
Answer:
Yes, ROS can also be used for real aircraft control. Pay attention to switching data sources when migrating code from simulation to real hardware.
8.3.11 TensorRT compilation always reports errors. Why?
Answer:
This is typically a library compatibility issue. The C++ compiled version may not match — the code may use features from C++11 or higher versions.
8.3.12 For two UAVs with onboard computers doing SIL simulation in one RflySim environment, must sensor parameters be configured?
Answer:
Yes. In addition to IP addresses, the two onboard computers need different sysid values. On the RflySim3D computer, the client config must include all cameras, and the server config must be configured separately for each aircraft. First determine how many aircraft you want in RflySim, then configure sensors according to each aircraft's ID.
8.3.13 Does the RflySim platform camera model support distortion parameters (e.g., principal point offset)?
Answer:
The RflySim camera model is an ideal pinhole model with no distortion. Even when calibrated using a calibration board, the intrinsic parameter matrix will be close to ideal values.
8.3.14 Can annotated source code for the ring-traversal, ball-strike, and face recognition demos be provided?
Answer:
Chapter 8 PPT Section 3 includes code annotations. Path:
[RflySim Installation Directory]\RflySimAPIs\8.RflySimVision\PPT.pdf
8.3.15 How realistic is the simulation environment (gravity, magnetic field, air pressure, physical engine, sensors)?
Answer:
Gravity, magnetic field, and air pressure models use mature, internationally standardized Simulink modules. The aerodynamic drag model is fitted from real flight data and requires experimental calibration for fidelity. Sensor models also use established models that can be verified by comparing experimental and simulation data.
8.3.16 What are the three components in the Python vision ring-traversal routine?
Answer:
This routine implements ring traversal remotely. config is the configuration file, client runs on Windows, and server runs on Linux. Reference:
[Installation Directory]\RflySimAPIs\8.RflySimVision\1.BasicExps\1-VisionCtrlDemos\e5_ScreenCapAPI\2-CrossRing\
8.3.17 With two drones doing visual-collaborative algorithms, the first aircraft's camera perspective works but the second's is empty. How to configure the second aircraft's camera?
Answer:
The JSON file parameters are not set correctly. The SeqID values must be different — set one to 0 and the other to 1.
8.3.18 Are there tutorials for porting visual perception and control to an onboard computer (semi-physical)?
Answer:
This tutorial is being refined. The latest platform version supports the robot race scenario:
C:\PX4PSP\RflySimAPIs\8.RflySimVision\1.BasicExps\2-BaseDemoAuto
The recommended workflow:
- Develop algorithms in WinWSL (Ubuntu 20.04), subscribe to ROS image topics from RflySim, and control the aircraft via mavros.
- Copy the algorithm to the real onboard computer, apply GPU acceleration/optimization, and perform onboard HIL simulation.
- Switch the ROS image subscription source from simulation to the real camera and calibrate with actual camera parameters.
- Perform real flight testing.
A more detailed tutorial is planned for release with version 3.05.
8.3.19 For current UAV navigation — is visual or LiDAR-based positioning more stable and reliable?
Answer:
For indoor environments, LiDAR is recommended. For low-altitude outdoor environments, the choice depends on the application scenario, but LiDAR is generally recommended for its overall reliability.
8.3.20 How to change the camera to a body-fixed mount in the visual routine?
Answer:
Check the sensor configuration in the API documentation. The MountType setting controls mounting mode. The RflySim Toolchain's gimbal functionality has also been developed and can be used.
8.3.21 When the camera is fixed to the body, does the camera angle tilt with the aircraft during forward flight?
Answer:
The RflySim Toolchain supports two modes: one that follows the aircraft's tilt angle, and one that is independent of the aircraft's tilt angle.
8.3.22 For unmanned system vision development, which is more accurate for image recognition — YOLO or large language models?
Answer:
YOLO is more accurate for image recognition tasks.
For more questions, please visit: https://github.com/RflySim/Docs/issues