ScreenCapApiV4 Interface Documentation

Introduction

Overview: This module provides a Python interface for multi-window capture and manipulation on the Windows platform. It supports retrieving window handles, extracting window content into OpenCV-format images, and moving specified windows, meeting the requirements for screen content acquisition in multi-window scenarios.

In RflySim drone simulation tasks, simulation visualization windows and various task debugging windows often coexist. Many vision-based autonomous drone tasks require directly capturing real-time frames from simulation windows rather than acquiring them through camera sensor channels. This module adapts to the Windows windowing mechanism, supporting simultaneous enumeration and capture of multiple target windows, and outputs images in a format compatible with OpenCV processing, facilitating developers to directly integrate with various vision detection and recognition algorithms. It is commonly used in RflySim platform computer vision workflows for screen capture in multi-window simulations, simulation demo layout adjustments, and similar tasks.

Quick Start

The following example retrieves the handle and window information of the first RflySim3D window, then continuously captures and displays its frames.

Reference example: [RflySim installation path]\RflySimAPIs\8.RflySimVision\1.BasicExps\1-VisionCtrlDemos\e5_ScreenCapAPI\1-ShootBall

import cv2
import ScreenCapApiV4 as sca


windowHandles = sca.getWndHandls()
if not windowHandles:
    raise RuntimeError("RflySim3D window not found")

windowInfo = sca.getHwndInfo(windowHandles[0])

while True:
    imageBgr = sca.getCVImg(windowInfo)
    cv2.imshow("RflySim3D", imageBgr)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

Environment & Dependencies

• Python Environment: >= 3.8.10
• Dependencies: ctypes, cv2, d3dshot, numpy, sys, win32con, win32gui, win32ui
• Prerequisites: Before calling this interface, ensure the system supports screen capture functionality and that the RflySimSDK.vision module has been correctly imported.

Core Interface Description

The module ScreenCapApiV4.py includes configuration variables, helper functions, and core business classes.

Global Constants and Enumerations

This section lists all globally accessible constants and enumeration definitions directly referenceable within the module.

Standalone Constants

None

---

Global/Standalone Functions

window_enumeration_handler(hwnd, window_hwnds)

Function Description: A callback function for window enumeration, used to collect handles of all top-level windows during Windows window enumeration. Parameters:

• hwnd: Handle of the window currently being enumerated
• window_hwnds: List container used to store collected window handles

Return Value:

• None (returning a non-zero value indicates continuing enumeration)

Exceptions: None

---

getWndHandls()

Function Description: Enumerates all top-level windows currently present in the system and collects their window handles. Parameters: None
Return Value:

• list[int]: A list of handles for all top-level windows in the system

Exceptions: None

---

getHwndInfo(hWnd)

Function Description: Retrieves basic information (e.g., position and size) of a specified window based on its handle, for subsequent window screenshot operations. Parameters:

• hWnd: Handle of the target window

Return Value:

• dict: A dictionary containing window handle, position coordinates, and size dimensions

Exceptions: None

---

getCVImg(wInfo)

Function Description: Captures a screenshot of the specified window and converts it into an OpenCV-compatible BGR image format. Parameters:

• wInfo: Window information dictionary containing window handle, position, and size, obtained via getHwndInfo

Return Value:

• numpy.ndarray: BGR screenshot array in OpenCV format

Exceptions: None

---

getCVImgList(wInfoList)

Function Description: Performs batch screenshot capture of multiple windows and converts them into OpenCV format images. Parameters:

• wInfoList: A list of window information dictionaries, each representing a single window

Return Value:

• list[numpy.ndarray]: A list of OpenCV-format screenshots corresponding to each window

Exceptions: None

---

moveWd(hwd, x=0, y=0, topMost=False)

Function Description: Moves the specified window to a given screen coordinate position, optionally setting it to always stay on top. Parameters:

• hwd: Handle of the target window
• x: Target X-coordinate of the window’s top-left corner on the screen (default: 0)
• y: Target Y-coordinate of the window’s top-left corner on the screen (default: 0)
• topMost: Whether to set the window as always-on-top (default: False, i.e., not always on top)

Return Value:

• None

Exceptions: None

---

clearHWND(wInfo)

Function Description: Releases GDI resources occupied during window screenshot operations to prevent resource leaks. Parameters:

• wInfo: Window information dictionary that has completed screenshot operations and contains GDI resource handles requiring release

Return Value:

• None

Exceptions: None

---

WinInfo Class

Stores resources and dimensional information related to Windows window screenshot operations, providing foundational data structures for screen capture functionality.

__init__(hWnd, width, height, saveDC, saveBitMap, mfcDC, hWndDC)

Function Description: Initializes a window information object, storing various resource handles and dimensional parameters required for window screenshot operations. Parameters (Args):

Parameter Name	Type	Required	Default	Description
hWnd	int	Yes	-	Handle of the target window
width	int	Yes	-	Width of the capture area (in pixels)
height	int	Yes	-	Height of the capture area (in pixels)
saveDC	int	Yes	-	Handle of the compatible device context
saveBitMap	int	Yes	-	Handle of the bitmap object
mfcDC	int	Yes	-	Handle of the MFC device context
hWndDC	int	Yes	-	Handle of the target window's device context

Return Value (Returns):

• WinInfo instance object

Exceptions (Raises):

• None

---

Advanced Usage Example

The following example retrieves two RflySim3D windows, adjusts their screen positions, and uses getCVImgList to batch capture images from both windows.

Reference example: [RflySim installation path]\RflySimAPIs\8.RflySimVision\1.BasicExps\1-VisionCtrlDemos\e5_ScreenCapAPI\2-CrossRing

import cv2
import ScreenCapApiV4 as sca


windowHandles = sca.getWndHandls()
if len(windowHandles) < 2:
    raise RuntimeError("This example requires two RflySim3D windows")

# Place the two windows side by side
nextX, nextY = sca.moveWd(windowHandles[0], 0, 0, True)
sca.moveWd(windowHandles[1], nextX, 0, False)

windowInfoList = [
    sca.getHwndInfo(windowHandles[0]),
    sca.getHwndInfo(windowHandles[1]),
]

while True:
    imageList = sca.getCVImgList(windowInfoList)
    for index, imageBgr in enumerate(imageList):
        cv2.imshow("RflySim3D-" + str(index), imageBgr)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

Notes and Pitfall Avoidance Guide

• Window Validity Validation: Before calling screenshot-related methods of the WinInfo class, you must first invoke the is_window_valid() method to verify that the target window exists and is capturable. If the simulation window is closed or minimized to the background, skipping this validation will directly cause the program to throw a null pointer exception.
• Multiple Window Instance Conflicts: Do not instantiate multiple WinInfo objects for the same simulation window. Repeated instantiation will cause the window device context resources to be occupied multiple times, ultimately leading to screenshot lag or even program crashes.
• Asynchronous Task Resource Limitations: When performing batch asynchronous screenshots, it is recommended not to process more than 8 windows simultaneously. Excessive concurrent tasks will consume a large amount of GPU memory and CPU resources, causing the simulation program's frame rate to drop or screenshot frames to be lost.
• Window Title Matching Rules: When instantiating WinInfo with a fuzzy window name, it will by default match the first window that meets the name characteristics. If multiple simulation windows with the same name exist, it is recommended to use the full window title to avoid matching errors.

Changelog

• 2024-08-05: fix: Added HTML version API comments
• 2024-07-17: fix: Updated VisionCaptureApi interface API
• 2023-10-23: feat: Add all Python common labs