Chapter 3: 3D Environment Modeling and Simulation¶
This chapter primarily introduces how to build realistic 3D and physical environment simulations based on Unreal Engine (UE4 / UE5). The learning focus centers on the working principles of RflySim3D and RflySimUE5 engines, the customized import process for 3D vehicles and scene models, and ultimately achieving visual closed-loop control, complex climate switching, and even ultra-large-scale global terrain construction based on GIS and Cesium.
3.1 Background and Theory¶
RflySim3D is the core engine within the RflySim system responsible for visual rendering and high-fidelity sensor physical feedback. In advanced development scenarios for unmanned systems—such as machine vision (e.g., SLAM, deep reinforcement learning), infrared sensing, and LiDAR-based point cloud obstacle avoidance—the purely dynamical computation (e.g., provided by CopterSim) cannot satisfy the data requirements of the perception layer; thus, a highly realistic 3D physical world simulation becomes essential.
The engine acquires the six-degree-of-freedom (6-DOF) pose data of the aircraft, computed by the autopilot, via a UDP communication architecture and maps this data onto the virtual scene. Internally, it deeply integrates real-time ray tracing, customizable artificial rain and particle system effects, and supports high-flexibility XML parameter configuration—covering all aspects of perception environment simulation, from material lighting to weather and time evolution.
3.2 Framework and Interfaces¶
Building one’s own interactive 3D scenes often involves certain barriers related to artistic design and C++ programming. To lower the development threshold, RflySim has developed a modular 3D interface and a precompiled component mounting logic system.
3.2.1 Engine Communication and Control Model¶
At the framework level, RflySim3D communicates with external computation systems using an extremely lightweight local area network multicast UDP or shared memory protocol, enabling zero-copy integration. Developers can not only issue control commands directly via Python/Simulink, but also utilize predefined XML nodes (e.g., RefPoints) to determine default terrain coordinates, aircraft skins, default sensor offset orientations, and even obstacle matrices.
3.2.2 High-Fidelity Perception and Specialized Component Library¶
The system’s underlying interface supports a full suite of sensor ecosystems, including RGB cameras, global depth maps, semantic segmentation masks, and 360° LiDAR (with multi-line microsecond-level scanning echo simulation). Additionally, it provides prebuilt modules such as infrared pod lock-on guidance effects and multi-aircraft formation trail collision effects. These can be directly exposed to ROS via the built-in Visual/AirSim mapping interface.
3.2.3 Wide-Area GIS and Physical Extension Integration¶
For large-scale simulation tasks, the toolchain includes a Cesium-based loading interface and associated plugins. Developers can use this interface to dynamically stitch oblique photogrammetry models, high-resolution satellite imagery, or elevation terrain data (DEM) into the UE map in real time. Coupled with the dynamic weather module library and mechanisms such as V-REP-based physical gravity collision skeleton import, this enables the creation of the most realistic geographical confrontation environments for unmanned clusters.
3.3 Showcase of Outstanding Cases¶
3.4 Course-Linked Video Resources¶
Public Lecture Replay (Session 3: In-Depth Development of 3D Scenes):
3.5 Chapter Experiment Cases¶
The verification experiments and guided cases related to this chapter are located in the [Installation Directory]\RflySimAPIs\3.RflySim3DUE folder.
3.5.1 Interface Learning Experiments¶
Located in the 3.RflySim3DUE\0.ApiExps folder, covering foundational interface tutorials for the platform and general introductions to various tools.