Drone Control Using Upper Body Gesture Based on Pose Detection and Random Forest for Monitoring in Smart Agriculture

Sustainable development goals (SDGs) have become a global concern. As the second target of the SDGs [1], zero hunger needs to be addressed by improving performance in sustainable agriculture [2, 3]. Agricultural activities with priority services are rice, corn, soybean. Corn is one of the priority crops with high productivity and various advantages, such as for feed, food, energy, and industrial raw materials [4, 5]. Agricultural sustainability can be achieved by synergizing agriculture and technology through smart agriculture. Smart agriculture involves the use of sensors and actuators, information and communication technologies (ICT), internet of things (IoT), and artificial intelligence robots to support agriculture [6, 7] in soil preparation, seed planting, irrigation [8], crop cultivation, monitoring, harvesting, and logistics [9]. Monitoring is the third ranking agricultural robot function with a value of 13.8% below weeding and tool-carrier [10]. The urgency of this study is to develop the ability to control the drone to monitor the corn plant disease to support smart agriculture for increasing crop yields.

Corn growth was first monitored using radar by Ulaby and Bush in 1976 [11]. The classification of corn leaf disease was investigated by using MobileNetV2 Convolutional Neural Network (CNN) architecture based on images taken manually [12]. A wheeled mobile robot was remotely controlled by using a joystick based on data from the Global Positioning System (GPS) [13]. Some previous researches [14, 15] studied drone’s control and guidance that can be used to support smart agriculture. Both papers [14, 15] focus on hardware-centric enhancement and software-driven autonomy, respectively, for drone control. However, neither addresses control paradigms involving direct, intuitive human-machine interaction like gesture-based control. Remote control was used to control an agricultural robot to perform several agricultural duties [16]. An unmanned aerial vehicle (UAV) in the form of quadcopter, controlled by a device with a touch-screen and a graphical user interface (GUI). To increase the transmission range, this UAV was completed with a repeater. Remotely controlled UAVs that follow the desired path and avoid obstacles by exploiting aerial and satellite data were used to modernize natural forest management by supporting monitoring activities [17]. Crop monitoring for plant disease detection was carried out by drone [18], including spraying and assessing fruit color and ripeness. A combination of global and local information was applied in a drone to detect and count maize tassel in complex agricultural [19]. Remote control can be alternatively substituted by exploiting human body gestures to control the robot intuitively. The study on gesture-based control (GBC) has an impact on drone monitoring in smart agriculture and it also increases the need for more intuitive and efficient human-robot interaction (HRI) systems. A gesture-based control for communicating with a robot using gesture recognition trained by a hand dataset was studied by Peral et al. [20]. The robot did not use joystick or glove as external and wearable devices for remote control. The gesture recognition proposed by Peral et al. [20] was based on deep learning on key frames of pose-detected hand images. The training and testing sets were divided into 74% and 26%, respectively. It resulted in 3117 training and 1101 testing gesture instances. For validation purposes, 10% of all images in the dataset were selected, resulting in 422 validation images. The IVO robot was used in the experiments. It was equipped with a depth sensor set as the onboard camera. However, the gestures were not implemented as commands to make the IVO robot move. The focus was on validating recognition and interaction capability, not gesture-based motion control. In contrast to previous research [20], this research focuses on developing gesture recognition to generate drone movement commands. Body gestures are a type of nonverbal communication that has more variety than hand gestures in expressing a message. It has the potential to deliver information from humans to robots in the form of movement control commands to improve interaction. However, accurately and actually interpreting information from gestures requires a combination of appropriate technological approaches. To address this challenge, this study aims to develop gesture-based control of drone monitoring using human body gestures.

This study proposes a gesture recognition system for human upper body gestures by combining pose detection and random forest (RF) algorithm. The key points of the human upper body are identified using pose detection, which is carried out using a machine learning model. The extracted features are then classified using RF.

This study makes three contributions. First, we developed an RGB image-based human upper body gesture recognition system to control a drone for a monitoring task by tracking 33 human body key points by exploiting the MediaPipe pose detection and gesture classification by utilizing RF algorithm. This approach has the advantage that a human operator controls the drone’s movement intuitively without additional or wearable hardware such as a joystick or dedicated gloves with sensors. Second, this study generated a dataset with 6,000 images representing 10 gestures for controlling the drone movement such as up, down, forward, backward, left, right, turn left, turn right, hover, and stop. The proposed gesture recognition method was trained and tested on the dataset by randomly splitting it into 80% and 20%. Third, the proposed method was implemented in real-time gesture recognition for controlling drone movement in simulations and real-world experiments. This proposed gesture-based control system was also tested for controlling the movement of drones to support smart agriculture in a real environment.

This section describes the steps in developing a new system for gesture recognition based on the proposed method, which comprises of feature extraction using pose detection and gesture classification using RF algorithm.

3.1 Environment setting and dataset development

The initial stages in developing a gesture recognition system for pose detection-based drone control include setting the environment and compiling the dataset. Setting the environment prepares the working environment for dataset collection, as shown in Figure 1.

Figure 1. Environment settings for the preparation of gesture data capture

Figure 2. Flowchart of dataset development

Based on the environmental settings, a white wall from the automation and robotics systems laboratory without a green screen was used as the background. The camera and the group members performing the gestures are separated by approximately 1.43 m. The internal webcam camera is located 1.21 m above the floor. The positions of users are marked with red boxes, and the camera is marked with a green box. The dataset development process is shown in Figure 2. The image resolution of the dataset is 640 × 480. The number of datasets is 600 poses for each gesture. There are 10 folders representing 10 gestures to save this dataset. The dataset contains 6,000 images representing 10 gestures.

These images were recorded from 5 participants demonstrating 10 gestures in front of the camera. The proposed pose detection system uses MediaPipe. MediaPipe is a pose landmark that marks objects with nodes. MediaPipe tracks 33 body landmark features, representing the approximate location of body parts within the human skeleton.

The MediaPipe nodes represent 33 positional landmark features on the human body, with x as the horizontal coordinate, y as the vertical coordinate, z as the relative depth, and visibility. There are 33 nodes (vertices) connected by some edges to form an undirected graph of the full human pose as the result of pose detection in this study exploited MediaPipe BlazePose [26], as shown in Figure 3.

Figure 3. Nodes or landmark features in the human upper body

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 4. Gesture design to control drone movement represents ten motions consisting of (a) up, (b) down, (c) forward, (d) backward, (e) left, (f) right, (g) turn left, (h) turn right, (i) hover, and (j) stop

The pose detection-based gesture recognition system for controlling drone uses the upper body command gesture with 25 landmark nodes. From the head, hands, and body to the human waist. The planned gesture material is represented as a human skeleton used in the upper body gesture recognition system, as shown in Figure 4. This process produces a set of gesture data called a gesture dataset in the dataset compilation stage.

The gesture data are stored in several folders with label names according to the gesture. Each folder represents one gesture category: up, down, forward, backward, left, right, turn left, turn right, hover, and stop. The storage folder is divided into each gesture category for the next stage of data processing. Figure 5 shows samples of ten gestures of the dataset with 6,000 images. Each of the 10 gestures have 600 images.

Figure 5. Dataset with 6,000 images representing ten gestures

3.2 Feature-extraction using pose detection

The dataset was extracted using pose detection, and the results were saved into a comma separated value (csv) file. This step uses MediaPipe and the csv library. MediaPipe is initialized in the still image mode with a minimum detection confidence level of 30%. Pose detection detects 33 landmark features of the human pose. Each body feature comprises four data: x, y, z, and visibility. The feature extraction process is shown in Figure 6. This extraction results in 132 values (33 points × 4 values per point) per upper body image, as shown in Figure 7.

Figure 6. Flowchart of feature extraction using pose detection

3.3 Training and testing based on random forest algorithm

The RF architecture involves a combination of multiple decision trees [27] built using a training dataset of the pose detection dataset organized into bootstrap datasets. The RF architecture is depicted in Figure 8. Each decision tree is trained with one bootstrap dataset, and the classification results from each tree are voted. The majority vote from all decision trees determines final classification result, which becomes the output of this algorithm.

Figure 7. Pose detection extracts nodes from the user’s upper body image

Figure 8. Random forest (RF) architecture

Figure 9 depicts the decision trees in the RF. The dataset is the root node as the decision tree’s starting point of the decision tree. The detected dataset will be evaluated, for example, "is the user’s hand stretched to the side or in a ready standing position?".

The results of the branching will be evaluated in the next stage, for example, "is the user doing a straight hand pose to the left?". The final result or decision is determined by the last node. For example, the result of the gesture is “turning left” as the command output to control the robot’s movement. The training flowchart is depicted in Figure 10.

The CSV files containing body pose data were combined into one large CSV file for RF training. CSV of all images in the dataset is read and separated into feature and label data. The feature and label data are randomly partitioned into training and testing data with ratios 80% and 20%, respectively.

Figure 9. Decision tree in random forest architecture

Figure 10. Flowchart of training data using random forest (RF)

An RF is prepared by defining the number of decision trees in the forest. In this study, the default value is 100. The bootstrap is created by maintaining the randomness of the samples used when building trees using the value of 42. The RF algorithm is trained using feature data paired with target labels to produce a training model. The model file is used in the testing process, as shown in Figure 11.

Figure 11. Flowchart of test for the gesture-based control (GBC) system

The testing process involves loading the model and preparing the camera to capture the user’s actual RGB image. RGB images are used in the pose detection process. The extracted landmark features are compared with the training model in the gesture prediction process to obtain the detected gesture output. The visualization of landmark pose detection was successfully performed using the MediaPipe library, which identified 33 body points and displayed the gesture classification results directly on the screen, as shown in Figure 12. The system detected the "hover" gesture by labeling it "Pose: hover" and mapped the body position with a skeleton line based on the detected landmarks.

Figure 12. Gesture recognition testing using pose detection and random forest (RF) algorithm

3.4 Drone control system

Figure 13 shows the gesture-based drone control system. The RF model $\text{FL}_{xy}^{m}$ resulted from training process is employed as the ground truth that acts as a reference for robot movement. The detected landmark features $\text{F}_{xy}^{a}$ from the user who demonstrate their gestures in front of the camera are used to predict the gesture command ${{G}_{c}}$ based on the model $\text{FL}_{xy}^{m}$.

Figure 13. Block diagram of the gesture-based drone control system

The drone motion is described as follows [28]:

$\left[\begin{array}{l}U_1 \\ U_2 \\ U_3 \\ U_4\end{array}\right]=\left[\begin{array}{cccc}k_t & k_t & k_t & k_t \\ 0 & -l k_t & 0 & l k_t \\ k_t & 0 & k_t & 0 \\ k_d & -k_d & k_d & -k_d\end{array}\right]\left[\begin{array}{c}\omega_1^2 \\ \omega_2^2 \\ \omega_3^2 \\ \omega_4^2\end{array}\right]$         (1)

$\left[\begin{array}{c}\ddot{x} \\ \ddot{y} \\ \ddot{z}\end{array}\right]=\frac{1}{m}\left(\left[\begin{array}{c}\left(U_1\right)(c \psi s \theta c \phi+s \psi s \phi) \\ \left(U_1\right)(s \psi s \theta c \phi-c \psi s \phi) \\ \left(U_1\right)(c \phi c \theta)\end{array}\right]-\left[\begin{array}{c}0 \\ 0 \\ m g\end{array}\right]\right)$         (2)

where, ${{U}_{1}}$ describes the total thrust of all motors, ${{U}_{2}}$ expresses roll control input, ${{U}_{3}}$ represents pitch control input, ${{U}_{4}}$ presents yaw control input, ${{k}_{t}}$ symbolizes thrust coefficient, ${{k}_{d}}$ symbolized drag torque, $l$ is length of arm, and $\omega $ represents angular velocity of each motor, $m$ describes the mass, $g$ expresses the gravity acceleration. While $s$ and $c$ are abbreviation of sine and cosine of roll $\phi $, pitch $\theta $, and yaw $\psi $ angle, respectively. The output of the equation of motion is $\left[\begin{array}{lll}\ddot{x} & \ddot{y} & \ddot{z}\end{array}\right]^T$ the drone acceleration along the X, Y, and Z axes.

Based on (1) and (2), the controller generates velocity commands $\dot{\mathrm{q}}^c$ based on the detected gesture command ${{G}_{c}}$. The drone moves by using $\dot{\mathrm{q}}^c$ to reach new position $\text{P}$ in environment. The following are the mapping between gesture commands ${{G}_{c}}$ to drone control inputs as position commands.

${{U}_{1}}={{k}_{t}}\left( \omega _{1}^{2}+\omega _{2}^{2}+\omega _{3}^{2}+\omega _{4}^{2} \right)$          (3)

$\overset{\ddot{\ }}{\mathop{z}}\,=\frac{1}{m}\left( \left( {{U}_{1}} \right)\left( c\phi c\theta  \right)-mg \right)$        (4)

$\overset{\ddot{\ }}{\mathop{x}}\,=\frac{1}{m}\left( \left( {{U}_{1}} \right)\left( c\psi s\theta c\phi +s\psi s\phi  \right) \right)$           (5)

$\overset{\ddot{\ }}{\mathop{y}}\,=\frac{1}{m}\left( \left( {{U}_{1}} \right)\left( s\psi s\theta c\phi -c\psi s\phi  \right) \right)$          (6)

${{U}_{4}}={{k}_{d}}\left( \omega _{1}^{2}-\omega _{2}^{2}+\omega _{3}^{2}-\omega _{4}^{2} \right)$        (7)

$\overset{\ddot{\ }}{\mathop{\psi }}\,=\frac{{{U}_{4}}}{{{I}_{Z}}}$         (8)

The gesture commands of upward, downward, and hover are related with (3) and (4). To move the quadcopter to upward direction, the angular velocity of all four motors should be increased equally to overcome gravity. To move the quadcopter to downward direction, the angular velocity of all four motors should be decreased equally. To hover, all four motors should spin at an equal speed such that total upward thrust equals the force gravity.

The gesture commands of forward and backward are mapped to (3) and (5). The forward motion is achieved by speed up rear motors and slow down front motors. On the contrary, the backward motion is realized by speed up front motors and slow down rear motors.

The gesture commands of left and right motion are applied by utilizing (3) and (6). The left motion is reached by speed up right-side motors and slow down left-side motors. Whereas, the right motion is attained by speed up left-side motors and slow down right-side motors.

The gesture commands of rotate left and rotate right motion are implemented by using (7) and (8). Rotate left is achieved by increasing speed of clock wise (CW) motors (motor 1 and 3) and decreasing speed of counter clock wise (CCW) motors (motor 2 and 4). On the other hand, rotate right is reached by increasing speed of CCW motors and decreasing speed of CW motors. Table 1 shows the specifications of some of the software used for the proposed system implementation.

Table 1. Specification of the software used in this study

Drone control using upper body gestures based on pose landmark detection and the RF algorithm was successfully implemented. The program could recognize upper body gestures in real-time through the camera and convert them into control commands sent to the robot. MediaPipe pose detection was effective in detecting and extracting body landmark points, which were then processed into numerical features for the classification.

The average accuracy of gesture recognition on ten gesture commands performed without any pre-processing is 89.1%. By employing the augmented dataset and utilizing the hip-center normalization, the proposed RF-based method demonstrates a strong balance between high classification accuracy 99.70%, moderate computational cost 18.35 ms, real-time capability 26.68 ms, and end-to-end latency 56.33, outperforming both SVM and 1D-CNN in terms of overall suitability for practical gesture recognition systems.

These results validate the effectiveness of the proposed approach for real-time applications, particularly in scenarios where computational resources are limited and reliable gesture recognition is critical. The experiment results show that the robot can move appropriately according to gesture-based commands such as up, down, forward, backward, left, right, turn left, turn right, hover, and stop. Based on the simulation and experimental results, it was concluded that the proposed system can be used as an alternative interactive robot control method. In future research, we will investigate gesture recognition to control multiple drones to support smart agriculture.