AutoMavDB Interface Documentation¶

Introduction¶

Overview: This file provides capabilities for creating, registering, and reading data from a PX4 flight controller log database, enabling structured storage and convenient querying of PX4 drone flight logs.

This module is a structured management tool designed by the RflySim simulation platform specifically for PX4 flight controller log data. It organizes the various types of flight data output by the PX4 flight controller into a standardized database structure, facilitating subsequent analysis and processing of simulation or real-flight data. It is applicable to scenarios such as drone flight mission debriefing, control algorithm performance validation, and flight data statistical analysis, helping users quickly extract target data from raw logs and eliminating the cumbersome manual log整理 process.

The DataREG class handles flight data item registration, the MAVDB class manages the complete log database, and the DATAAPI class provides a unified data query interface. Coupled with the helper dict_factory for converting database row data into dictionary format, this forms a complete and user-friendly solution for managing PX4 log data, supporting developers in flexibly calling and retrieving various flight parameters.

Quick Start¶

A minimal runnable example; copy and modify only the minimum required configuration to run.

from RflySimSDK.phm.AutoMavDB import DATAAPI

# 1. Initialize DATAAPI, specifying the case ID; other parameters can use defaults
case_id = 1  # Modify to your own experiment case ID
data_api = DATAAPI(CaseID=case_id)

# 2. Get the DP log file path
dp_path = data_api.DP()
print(f"DP log file path: {dp_path}")

# 3. Get the MKD log file path
mkd_path = data_api.MKDPath()
print(f"MKD log file path: {mkd_path}")

# 4. Get the PX4 log file path
px4_path = data_api.PX4Path()
print(f"PX4 log file path: {px4_path}")

Environment and Dependencies¶

Python Environment: >= 3.8.10
Dependencies: AutoREG, QGCCtrlAPI, csv, fnmatch, json, lxml.html, matplotlib.pyplot, numpy, os, pandas, re, shutil, sqlite3, sys, threading, time
Prerequisites: Prior to invoking this interface, RflySimSDK initialization must be completed, and the relevant MAVLink data communication environment must be ready.

Core Interface Description¶

The module AutoMavDB.py contains configuration variables, helper functions, and core business classes.

Global Constants and Enumerations¶

This section lists all globally accessible constants and enumerations defined within the module.

Standalone Constants¶

None

Enumerations¶

`DataREG`¶

Name	Type	Value	Description
`lock`	`str`	`threading.Lock()`	-
`isDeleted`	`bool`	`False`	-
`MavNum`	`int`	`0`	-
`Mode2mavP`	`unknown`	`[]`	-
`Mode2caseP`	`unknown`	`[]`	-
`Mode3frameP`	`unknown`	`[]`	-
`Mode3caseP`	`unknown`	`[]`	-
`Mode4frameP`	`unknown`	`[]`	-
`Mode4caseP`	`unknown`	`[]`	-
`Mode4mavREG`	`unknown`	`{'Quadcopter': [], 'Fixedwing': [], 'Vtol': []}`	-
`html_file_path`	`str`	`os.path.abspath(os.path.join(sys.path[0], 'data', 'TestInfo.html'))`	-
`TestResult_html_file_path`	`str`	`os.path.abspath(os.path.join(sys.path[0], 'data', 'TestResult.html'))`	-
`bk_path`	`str`	`os.path.abspath(os.path.join(sys.path[0], 'conf', 'material', 'bk3.png'))`	-
`bk_path`	`str`	`bk_path.replace('\\', '//')`	-
`bk_path2`	`str`	`os.path.abspath(os.path.join(sys.path[0], 'conf', 'material', 'bk2.png'))`	-
`bk_path2`	`str`	`bk_path2.replace('\\', '//')`	-
`dataPollNum`	`int`	`0`	-
`Finally_Path_for_Multi`	`unknown`	`None`	-

Global/Standalone Functions¶

`dict_factory(cursor, row)`¶

Function Description: Converts SQLite query result rows into dictionary format, where keys are column names and values are corresponding row data, facilitating access to query results by column name.
Parameters (Args):

cursor: SQLite database query cursor, used to retrieve column name information from results
row: Single-row result data retrieved from the query

Return Value (Returns):

dict: Dictionary with column names as keys and corresponding row data as values

Exceptions (Raises):
None

`MAVDB` Class¶

Manages database and JSON file interactions for drone fault injection test cases, supporting test case querying, status modification, and result storage. Suitable for data management in PHM (Prognostics and Health Management)-related testing workflows.

`init(conf)`¶

Function Description: Initializes the MAVDB database connection class.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`conf`	-	Yes	-	Database configuration parameters

Return Value (Returns):

MAVDB instance object

Exceptions (Raises):
None

`VISION()`¶

Function Description: No additional documentation provided.
Parameters (Args):
None
Return Value (Returns):

None

Exceptions (Raises):
None

`JSON_TO_SQL()`¶

Function Description: Synchronizes test cases from JSON files into the database.
Parameters (Args):
None
Return Value (Returns):

None

Exceptions (Raises):
None

`GET_CURSOR()`¶

Function Description: No additional documentation provided.
Parameters (Args):
None
Return Value (Returns):

None

Exceptions (Raises):
None

`GET_FAULT_CASE()`¶

Function Description: Retrieves fault test case information.
Parameters (Args):
None
Return Value (Returns):

Collection of fault test cases

Exceptions (Raises):
None

`GET_CASEINFO(case_id)`¶

Function Description: No additional documentation provided.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID

Return Value (Returns):

Detailed information of the specified test case

Exceptions (Raises):
None

`GET_CASEINFO_P(case_id, path)`¶

Function Description: No additional documentation provided.
Parameters (Args):

Parameter Name	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID
`path`	-	Yes	-	JSON file storage path

Returns:

Detailed information of the test case with the specified ID under the given path

Raises: - None

`MAV_JSONPro_P(case_id, path)`¶

Function Description: Modifies the test status information in a JSON file
Arguments:

Parameter	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID
`path`	-	Yes	-	Storage path of the JSON file

Returns:

None

Raises: - None

`GET_CASEID()`¶

Function Description: Retrieves the list of fault test case IDs
Arguments:
- None
Returns:

list: List of fault test case IDs

Raises: - None

`GET_MAVCMD(case_id)`¶

Function Description: Retrieves the sequence of control commands for a test case
Arguments:

Parameter	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID

Returns:

The sequence of control commands corresponding to the specified test case

Raises: - None

`RESULT_DBPro(data)`¶

Function Description: Processes the test result database by adding new test results
Arguments:

Parameter	Type	Required	Default	Description
`data`	-	Yes	-	Test result data to be stored

Returns:

None

Raises: - None

`TEST_STATEPro(case_id)`¶

Function Description: Processes the test case database to update the test status
Arguments:

Parameter	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID

Returns:

None

Raises: - None

`IS_TESTEDPro(case_id)`¶

Function Description: Determines whether a specified test case has been tested
Arguments:

Parameter	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID

Returns:

bool: True indicates the test case has been tested; False indicates it has not

Raises: - None

`RESETR_DB(case_id)`¶

Function Description: Processes the result database for repeated test cases by resetting the test results
Arguments:

Parameter	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID

Returns:

None

Raises: - None

`MAV_JSONPro(case_id)`¶

Function Description: Modifies the test status information in a JSON file
Arguments:

Parameter	Type	Required	Default	Description
`case_id`	-	Yes	-	Test case ID

Returns:

None

Raises: - None

Example:

from RflySimSDK.phm.AutoMavDB import MAVDB

# Initialize database connection
mav_db = MAVDB(conf={"host": "localhost", "db": "mav_test"})

# Synchronize JSON test cases into the database
mav_db.JSON_TO_SQL()

# Retrieve all fault test case IDs
case_id_list = mav_db.GET_CASEID()

# Retrieve the corresponding control commands for the first test case
if len(case_id_list) > 0:
    cmd_seq = mav_db.GET_MAVCMD(case_id_list[0])

`DATAAPI` Class¶

PX4 flight log data analysis API class, used for loading and managing flight log data, supporting reading and processing of mission flight data.

`init(CaseID, conf, Data=None, Info=None)`¶

Function Description: Initializes the PX4 flight log data analysis API object, loading flight log data and configuration information for the specified mission ID. Parameters (Args):

Parameter Name	Type	Default Value	Description
CaseID	`int`	-	Mission number, used to locate the corresponding log file
conf	`dict`	-	Configuration dictionary related to log reading, including log path and other configuration information
Data	`Any`	`None`	Preloaded flight data; if provided, existing data is used directly without re-reading logs
Info	`Any`	`None`	Flight mission-related information; if provided, existing mission information is used directly

Return Value (Returns): None
Exceptions (Raises): None

`DP()`¶

Function Description: Initializes data processing-related paths and retrieves the PX4 toolchain path
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`MKDPath()`¶

Function Description: Creates directory structure for storing logs and ground truth data
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`PX4Path()`¶

Function Description: Retrieves the absolute path of the PX4 toolchain
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`MFolder(TargetFilder_log, TargetFilder_truedata)`¶

Function Description: Manually sets target folder paths for storing logs and ground truth data
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
TargetFilder_log	*	Yes	None	Target folder path for log storage
TargetFilder_truedata	*	Yes	None	Target folder path for ground truth data storage

Return Value (Returns): None
Exceptions (Raises): None

`DataPro()`¶

Function Description: Downloads flight logs via the QGC ground station download interface and copies logs to the target log storage path
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`TruedataRecord()`¶

Function Description: Records ground truth data during simulation flight to file
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`InfoRecord()`¶

Function Description: Records basic aircraft information to file
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`Test_result_Record()`¶

Function Description: Records test results to file
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`CMDAna(cmd)`¶

Function Description: Executes command-line instructions and parses the output results
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
cmd	*	Yes	None	Command-line instruction string to be executed

Return Value (Returns): Processed output result of the command execution
Exceptions (Raises): None

`GetInfoP()`¶

Function Description: Reads and returns the currently stored basic aircraft information
Parameters (Args): None
Return Value (Returns): Tuple of basic aircraft information
Exceptions (Raises): None

`RName()`¶

Function Description: Renames and organizes stored log and data files
Parameters (Args): None
Return Value (Returns): None
Exceptions (Raises): None

`SExecute(start_dir)`¶

Function Description: Executes scripts or commands in the specified starting directory
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
start_dir	*	Yes	None	Starting working directory path for command execution

Return Value (Returns): None
Exceptions (Raises): None

`convert_newlines_to_br(text)`¶

Function Description: Converts newline characters in text to HTML line break tags <br>
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
text	str	Yes	None	Original text to be converted

Return Value (Returns): Converted text with HTML line break tags
Exceptions (Raises): None

`generate_html_with_style(html_table, bk)`¶

Function Description: Generates a styled HTML information page based on the input table content
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
html_table	str	Yes	None	HTML-formatted table content
bk	*	Yes	None	Background identifier or background configuration parameter

Return Value (Returns): HTML page string with complete styling
Exceptions (Raises): None

`generate_test_result_html_with_style(html_table, bk)`¶

Function Description: Generates a styled test result HTML page based on the input test result table content
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
html_table	str	Yes	None	HTML-formatted test result table content
bk	*	Yes	None	Background identifier or background configuration parameter

Returns: HTML page string containing the test results with full styling
Raises: None

`read_data_from_html_tolist(html_file)`¶

Function Description: Reads data from a specified HTML file and converts it into list format
Arguments:

Parameter Name	Type	Required	Default Value	Description
html_file	Any	Yes	None	Path to the HTML file to be read

Returns: List containing the data read and converted from the file
Raises: None

`generate_header(html_file, bk_path)`¶

Function Description: Generates the HTML header content for the data display page and writes it to the target HTML file
Arguments:

Parameter Name	Type	Required	Default Value	Description
html_file	Any	Yes	None	Path to the target HTML output file
bk_path	Any	Yes	None	Backup resource path, used to reference static resources required by the HTML

Returns: None
Raises: None

`generate_test_result_header(html_file, bk_path)`¶

Function Description: Generates the HTML header content for the test results page and writes it to the target HTML file
Arguments:

Parameter Name	Type	Required	Default Value	Description
html_file	Any	Yes	None	Path to the target HTML output file
bk_path	Any	Yes	None	Backup resource path, used to reference static resources required by the HTML

Returns: None
Raises: None

`add_br_to_html(new_html_data_list)`¶

Function Description: Adds line break tags to the HTML data list to optimize page display formatting
Arguments:

Parameter Name	Type	Required	Default Value	Description
new_html_data_list	Any	Yes	None	HTML data list to be processed

Returns: Processed data list with line break tags added
Raises: None

`to_html(dataInfo, html_file_path, bk_path)`¶

Function Description: Exports data information into an HTML-formatted display file
Arguments:

Parameter Name	Type	Required	Default Value	Description
dataInfo	Any	Yes	None	Data information to be exported
html_file_path	Any	Yes	None	Save path for the output HTML file
bk_path	Any	Yes	None	Backup resource path, used to reference static resources required by the HTML

Returns: None
Raises: None

`test_result_to_html(testresultdata, html_file_path, bk_path)`¶

Function Description: Exports test result data into an HTML-formatted test report file
Arguments:

Parameter Name	Type	Required	Default Value	Description
testresultdata	Any	Yes	None	Test result data to be exported
html_file_path	Any	Yes	None	Save path for the output HTML test report
bk_path	Any	Yes	None	Backup resource path, used to reference static resources required by the HTML

Returns: None
Raises: None

`add_br_to_test_result_html(new_html_data_list)`¶

Function Description: Adds line break tags to the test result HTML data list to optimize test report display formatting
Arguments:

Parameter Name	Type	Required	Default Value	Description
new_html_data_list	Any	Yes	None	Test result HTML data list to be processed

Returns: Processed test result data list with line break tags added
Raises: None

`add_CaseDescription_br(caseDescription, variables)`¶

Function Description: Adds line break tags to the case description content to adapt to HTML display formatting
Arguments:

Parameter Name	Type	Required	Default Value	Description
caseDescription	Any	Yes	None	Original test case description content
variables	Any	Yes	None	Variable information related to the test case

Returns: Case description content with line break tags added
Raises: None

`add_ControlSequence_br(testInfo)`¶

Function Description: Adds line break tags to the control sequence content to adapt to HTML display formatting
Arguments:

Parameter Name	Type	Required	Default Value	Description
testInfo	Any	Yes	None	Test information containing the control sequence

Returns: Processed test information with line break tags added
Raises: None

`add_TestResult_br(testresult)`¶

Function Description: Adds line break tags to test result content for HTML display compatibility.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
testresult	Any	Yes	None	Original test result content

Returns: Processed test result content with added line break tags.
Raises: None

`Get_test_result_frame(text)`¶

Function Description: Formats framework-type test results into HTML.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
text	Any	Yes	None	Original framework test result text

Returns: Formatted HTML content for framework test results.
Raises: None

`Get_test_result_shitl(text)`¶

Function Description: Formats hardware-in-the-loop (HITL) test results into HTML.
Parameters (Args):

Parameter Name	Type	Required	Default Value	Description
text	Any	Yes	None	Original HITL test result text

Returns: Formatted HTML content for HITL test results.
Raises: None

`InfoDown()`¶

Function Description: Downloads information data.
Parameters (Args): None
Returns: Downloaded information data.
Raises: None

Advanced Usage Example¶

Demonstrates complex collaborative scenarios (e.g., multi-class coordination, asynchronous control, batch operations).

The following example illustrates a complex scenario involving multi-class collaboration to batch import fault test data and generate standardized analysis results. It integrates MAVDB for database interaction and DATAAPI for path management and data preprocessing, enabling end-to-end automation from raw MAVLink JSON logs to fault case storage:

```python import RflySimSDK.phm as phm import json import os

Initialize core utility class instances¶

mav_db = phm.MAVDB() data_api = phm.DATAAPI()

Create output directory for batch test results¶

data_api.MFolder()

Batch process all raw JSON logs in the specified directory¶

json_log_dir = data_api.PX4Path() # Get default PX4 log storage path for log_file in os.listdir(json_log_dir): if not log_file.endswith('.json'): continue # Read raw JSON log with open(os.path.join(json_log_dir, log_file), 'r', encoding='utf-8') as f: raw_json = json.load(f) # Preprocess MAVLink data and parse flight controller commands processed_mav = mav_db.MAV_JSONPro_P(raw_json) cmd_list = data_api.CMDAna(processed_mav) # Convert data structure and insert into SQLite database insert_sql = mav_db.JSON_TO_SQL(processed_mav) cursor = mav_db.GET_CURSOR() cursor.execute(insert_sql) # Extract fault case information and store in result table fault_info = mav_db.GET_FAULT_CASE(case_id=mav_db.GET_CASEID(processed_mav)) result_data = mav_db.RESULT_DBPro(fault_info, data_api.DataPro(processed_mav)) # Asynchronously write test result records data_api.Test_result_Record(result_data)

Commit all database modifications¶

mav_db.GET_CURSOR().connection.commit()

Notes and Pitfall Avoidance Guide¶

Database Connection Lifecycle Management: After calling GET_CURSOR to obtain a database cursor, all batch operations must be manually committed before closing the connection at the end of the program. Uncommitted changes will not be persisted to the database file, and repeatedly opening unclosed connections may cause table-locking errors.
JSON Log Preprocessing Prerequisites: Before calling the MAV_JSONPro_P method to parse raw JSON logs, ensure the JSON structure conforms to the standard MAVLink message format. If the logs suffer packet loss or missing fields, filter out anomalies first using the DATAAPI.DataPro method; otherwise, subsequent database insertions will fail.
Storage Path Dependency Issues: Before calling path-related methods such as PX4Path or MKDPath, ensure the RflySim workspace directory has been correctly configured; otherwise, empty paths will be returned, leading to file read/write errors. If custom storage paths are used, manually invoke MKDIR to create any missing parent directories.
Memory Control for Batch Operations: When processing over a hundred log files in batch, avoid loading all logs into memory at once. Instead, adopt the per-file processing pattern shown in the examples—release the corresponding memory after each file is inserted into the database—to prevent out-of-memory errors caused by large log files.

Changelog¶

2024-09-24: feat: Added log recording functionality
2024-08-02: chore: Added code comments for generating HTML-format API documentation
2024-07-18: fix: Updated API homepage index
2023-11-09: commit by Jinhu Tu
2023-11-09: fix: Added interface class file for automated security assessment

AutoMavDB Interface Documentation¶

Introduction¶

Quick Start¶

Environment and Dependencies¶

Core Interface Description¶

Global Constants and Enumerations¶

Standalone Constants¶

Enumerations¶

DataREG¶

Global/Standalone Functions¶

dict_factory(cursor, row)¶

MAVDB Class¶

__init__(conf)¶

VISION()¶

JSON_TO_SQL()¶

GET_CURSOR()¶

GET_FAULT_CASE()¶

GET_CASEINFO(case_id)¶

GET_CASEINFO_P(case_id, path)¶

MAV_JSONPro_P(case_id, path)¶

GET_CASEID()¶

GET_MAVCMD(case_id)¶

RESULT_DBPro(data)¶

TEST_STATEPro(case_id)¶

IS_TESTEDPro(case_id)¶

RESETR_DB(case_id)¶

MAV_JSONPro(case_id)¶

DATAAPI Class¶

__init__(CaseID, conf, Data=None, Info=None)¶

DP()¶

MKDPath()¶

PX4Path()¶

MFolder(TargetFilder_log, TargetFilder_truedata)¶

DataPro()¶

TruedataRecord()¶

InfoRecord()¶

Test_result_Record()¶

CMDAna(cmd)¶

GetInfoP()¶

RName()¶

SExecute(start_dir)¶

convert_newlines_to_br(text)¶

generate_html_with_style(html_table, bk)¶

generate_test_result_html_with_style(html_table, bk)¶

read_data_from_html_tolist(html_file)¶

generate_header(html_file, bk_path)¶

generate_test_result_header(html_file, bk_path)¶

add_br_to_html(new_html_data_list)¶

to_html(dataInfo, html_file_path, bk_path)¶

test_result_to_html(testresultdata, html_file_path, bk_path)¶

add_br_to_test_result_html(new_html_data_list)¶

add_CaseDescription_br(caseDescription, variables)¶

add_ControlSequence_br(testInfo)¶

add_TestResult_br(testresult)¶

Get_test_result_frame(text)¶

Get_test_result_shitl(text)¶

InfoDown()¶

Advanced Usage Example¶

Initialize core utility class instances¶

Create output directory for batch test results¶

Batch process all raw JSON logs in the specified directory¶

Commit all database modifications¶

Notes and Pitfall Avoidance Guide¶

Changelog¶

`DataREG`¶

`dict_factory(cursor, row)`¶

`MAVDB` Class¶

`init(conf)`¶

`VISION()`¶

`JSON_TO_SQL()`¶

`GET_CURSOR()`¶

`GET_FAULT_CASE()`¶

`GET_CASEINFO(case_id)`¶

`GET_CASEINFO_P(case_id, path)`¶

`MAV_JSONPro_P(case_id, path)`¶

`GET_CASEID()`¶

`GET_MAVCMD(case_id)`¶

`RESULT_DBPro(data)`¶

`TEST_STATEPro(case_id)`¶

`IS_TESTEDPro(case_id)`¶

`RESETR_DB(case_id)`¶

`MAV_JSONPro(case_id)`¶

`DATAAPI` Class¶

`init(CaseID, conf, Data=None, Info=None)`¶

`DP()`¶

`MKDPath()`¶

`PX4Path()`¶

`MFolder(TargetFilder_log, TargetFilder_truedata)`¶

`DataPro()`¶

`TruedataRecord()`¶

`InfoRecord()`¶

`Test_result_Record()`¶

`CMDAna(cmd)`¶

`GetInfoP()`¶

`RName()`¶

`SExecute(start_dir)`¶

`convert_newlines_to_br(text)`¶

`generate_html_with_style(html_table, bk)`¶

`generate_test_result_html_with_style(html_table, bk)`¶

`read_data_from_html_tolist(html_file)`¶

`generate_header(html_file, bk_path)`¶

`generate_test_result_header(html_file, bk_path)`¶

`add_br_to_html(new_html_data_list)`¶

`to_html(dataInfo, html_file_path, bk_path)`¶

`test_result_to_html(testresultdata, html_file_path, bk_path)`¶

`add_br_to_test_result_html(new_html_data_list)`¶

`add_CaseDescription_br(caseDescription, variables)`¶

`add_ControlSequence_br(testInfo)`¶

`add_TestResult_br(testresult)`¶

`Get_test_result_frame(text)`¶

`Get_test_result_shitl(text)`¶

`InfoDown()`¶