Deep Reinforcement Learning and Model Predictive Control in Hybrid Deep Learning for Rubber Yield Forecast

Hevea brasiliensis (Pará rubber tree) is a significant industrial crop that occupies about 10 million hectares of the Earth's terrestrial surface [1], and generates over 11 million tons of natural rubber each year. For accurate forecast of rubber yield, the latex is often extracted repeatedly from the same rubber trees. If the interest lies in latex yield per harvest, it would be inevitable to collect data with the same measurement units from rubber trees across time. Considering the autocorrelation between rubber yields, mixed models must be adopted to process the sequentially correlated longitudinal data.

The rubber yield can be improved, if the weather is favorable, the vegetation is suitable, and the field conditions are close to the natural living environment. The ontology rubber model [2] offers a way to locate the suitable rubber cropland, in the light of every agro-climatic factor. With the aid of this model, it is possible to judge whether a field is suitable for rubber production, and predict rubber yield accurately, laying the foundation for sustained growth of rubber yield.

Deep neural networks (DNNs) help to predict and check yield, and identify the variation of rubber yield with genomes and weather conditions. As a kind of learning model, DNNs can automatically discover the underlying data representation, eliminating the need for manual feature input. A typical DNN consists of several nonlinear layers that convert the raw input into complex and sophisticated representations [3]. In this way, the system becomes deeper and more complex, and capable of obtaining precise results.

Khaki et al. [4] have hybridized convolutional neural network (CNN) with recurrent neural network (RNN) into CNNRNN model to predict crop yield based on environmental factors and management activities. The CNNRNN structure used to be introduced to derive rubber yield from three important influencing factors. The structure can capture the time dependency of environmental factors, shorten the computing time, and generalize the precise yield under uncertain conditions, without sacrificing the prediction accuracy.

In the CNNRNN model, the features are selected through error backpropagation, and the impact of each weather/soil condition is evaluated against yield. However, the RNN is prone to problems like vanishing or exploding gradients. During each iteration of training, the weight of each network node is updated according to the incomplete derivative of the error of the current weight. The gradient might be vanishingly small in some cases, making it impossible to modify the weight range. Exploding gradients are a problem that node weights are adjusted excessively during network training, due to the accumulation of a huge error gradient.

In this paper, the RNN in RNNCNN model is improved by recalling long-term inputs by long short-memory network (LSTM) cells, which are precisely constructed recurrent nodes that perform excellently in series modeling. In addition, a Q-learning-based deep reinforced learning (Q-DRL) system was employed in the RNN to realize efficient forecast of rubber yield with the best rewarding iterations. Then, a deep recurrent Q-network (DRQN) framework was built on Q-DRL for efficient crop prediction. Data parameters were fed into the sequentially stacked layers of CNNRNN. Based on the input variables, the Q-learning network generated a rubber yield forecast setting. Then, the CNNRNN results were mapped to Q-factors on a linear layer. After that, the rubber yield was estimated by including a mixture of parameterized attributes with a threshold into the reinforcement-learning agent. Eventually, the agent summarized the measures to eliminate errors, and evaluate forecast efficiency.

DRL-based RNN employs probabilistic analytical modelling schemes (e.g., data analytics and probabilistic bias-difference decomposition) to control the ambiguity in geometric estimations. In this paper, a model predictive control (MPC) was developed based on DRL probabilistic learning to integrate system ambiguity, fuse long-term forecasts, and reduce the effect of system error. After that, the MPC learns a probabilistic shift method utilizing Gaussian processes (GPs). Then, the MPC was applied to obtain a control series to minimize the predicted long-term cost, and realize real-time update of the learned model. In this way, our method, denoted as CNNRNNPCDRL, became more robust to model inaccuracies.

3.1 Data acquisition

This research aims to improve the prediction accuracy of rubber yield in Kerala, a southwestern coastal state of India. The potential factors affecting rubber yield, such as soil, rainfall, humidity, temperature, and average wind speed, were acquired from the India Metrological Department [10], and the data published by the Rubber Institute of India (RRII) [11], Kottayam. Multiple soil parameters were studied, including pH, organic carbon, phosphorus, potassium, calcium, iron, boron, etc.

To predict rubber yield, the data on the above factors from 2018 to 2021 were collected from different districts of Kerala, namely, Alappuzha, Ernakulam, Kannur, Idukki, Kasargode, Kollam, Kottayam, Kozhikode, Palakkad, Pathanamthitta, Thiruvananthapuram, Thrissur, and Wayanad.

3.2 Proposed CNNRNN Model

1.png

Figure 1. Structure of our CNNRNN model

The DRL has evolved over the time through data expansion and measurement improvement, as new ways emerge to determine, evaluate, and recognize the data process in rubber yield forecast frameworks. The prediction of rubber yield must fully consider the constantly changing factors. This paper initializes a CNN for the spatiotemporal changes of each feature: soil, rainfall, humidity, temperature, and average wind speed. These one-dimensional (1D) CNNs are abbreviated as s-CNN, r-CNN, h-CNN, t-CNN, and a-CNN, in turn. The high-level features extracted by the above CNNs were fused on a fully connected layer (FC) to reduce the dimensionality.

Based on the data on previous years (t-k, t-k+1, …, t-k, t), the RNN, which was augmented with CNN structure, was adopted to estimate the rubber yield of a certain state in year t. The mean yield data of all states in the same year, management data, and FC layer performance were imported to the cell, which derives the fundamental features processed by the said CNNs.

The DRL, Q-learning, and deep Q-network (DQN) frameworks for rubber yield forecast are detailed in this subsection. Figure 1 shows the structure of our CNNRNN model. The DRL is embedded in the structure to enhance the accuracy of yield prediction.

3.3 Reinforcement learning (RL)

As an AI paradigm, RL builds and trains strategies with an incentive and penalty system through adaptive programming [12]. During the RL, the problem is usually solved by Markov selection. The agent-like RL algorithm learns through cooperation and interaction with the surroundings. The agent would be rewarded for correct actions, and penalized for incorrect actions. The learning requires no manual intervention, for the agent automatically pursues a higher reward and a lower penalty. In our model, an agent in state ‘s’ (rubber yield prediction) takes an action ‘a’ (soil, rainfall, humidity, temperature, and average wind speed). After the action is completed, the agent receives a reward R(s, a), and enters a new state ‘s’. The mapping criteria between states and actions are called rules. In each state, a rule $\pi$ is calculated to define the agent’s action. Here, there are three possible states: low, medium, and high. Once a parameter value changes, the rule under which the crop yield can be predicted with the minimum error is considered the best rule for that parameter. Throughout its lifespan, the agent aims to find the best rule $\pi *$ that maximizes the overall low-cost incentive.

The best rule can be depicted as:

$\pi *(s)=\underset{a \in A}{\operatorname{argmax}} \sum_{s^{\prime} \in S} P_{s a}\left(s^{\prime}, a\right) V^{*}\left(s^{\prime}, a\right)$     (1)

For each state-action combination, a value factor $V^{\pi}(s, a)$ [13] was introduced to represent the forecast of the predicted incentive following $\pi$. The best rule, which is determined by the maximum incentive gained by the agent in each state, yields the optimal value factor:

$V^{*}(s, a)=R(s, a)$$+\max _{a \in A} \gamma \sum_{s^{\prime} \in s} P_{s a}\left(s^{\prime}, a\right) V^{*}\left(s^{\prime}, a\right)$     (2)

In this way, each RL agent is trained through the interaction with the surroundings. Through dynamic programming, every RL agent optimizes his/her benefits to find the best rule and value factor. Figure 2 illustrates the RL process.

2.png

Figure 2. RL process

3.4 Q-learning

The QL, as a way to determine the actions of an agent depending on a value-action factor, computes the probability for an action to occur in a given state. One of the most substantive advances in RL is the introduction of a non-political method called temporal variance management. To identify the most valuable action for a target policy, Q-learning assesses the condition-action range factor. With the current state (s) and action (a) as inputs, the Q function predicts the incentive, and provides arbitrary fixed values in the first step of configuration assessment.

3.5 DQN

DQN is an extended RL agent with DNN. It is comparable to a Q-table in QL, which records the mapping relationship between states and actions. Typical examples of DNNs capable of directly learning abstract representations from raw data include CNN, RNN, and sparse autoencoders. There is virtually no difference between a DQN agent and a Q-learning agent in the communication with the environment via a series of observations, actions, and benefits. The two serve the same task, for many annotations, activities and incentives adopt the DQN structure. During the improvement of the DQN agent, an understanding is randomly selected from storage to continue the process, and RNN is taken as the DQN to estimate the factor with weight $\theta$. As a result, the Q-network can be established by reducing the mean squared error (MSE) in the Bellman equation by adjusting variable $\theta_{i}$ in the i-th iteration.

The loss, i.e., the squared variance between the desired Q and the estimated Q, can be defined as:

$Loss=\left(r+\gamma_{a^{\prime}}^{\max } Q\left(s^{\prime}, a^{\prime} ; \theta^{\prime}\right)-Q(s, a ; \theta)\right)^{2}$     (3)

Gradient descent for the real variables is adopted to mitigate this loss.

3.6 DRL for CNNRNN

The DRL-based crop yield monitoring can be understood with the input parameter that converts supervised learning into RL. The surroundings need to be determined through prediction games, each of which consists of the mixture of parameterized attributes and a threshold. The attributes contribute to crop production. At the beginning of the game, the agent performs tasks to get rewards on the value of crop yield. If the value falls in the desired range, the agent would receive a positive incentive; otherwise, it will receive a negative incentive.

Normal RL techniques like QL cannot easily discriminate and analyze yield predictions, owing to their limitations in state definition. By virtue of the strength of DQN in big data analysis, this paper employs a CNNRNN to estimate rubber yield based on various factor, such as soil, rainfall, humidity, temperature, and average wind speed. The CNNRNN is sophisticated enough to mine spatiotemporal and lexical features in forecast, language modeling, and voice recognition. As a variant of artificial network, the CNNRNN combines the input of the current state with the output of the previous state to determine the results. Briefly speaking, the network can recall earlier data, and use them in the calculation of the current state.

The DQN agent was created by stacking the CNNRNN layers sequentially, initializing the network parameters through pre-learning, and adding a linear layer that maps the RNN results to $\mathrm{Q}$ scores. Let $x^{t}$ be the input of the learning sample at time $\mathrm{t} ; \mathrm{H}^{\mathrm{t}}$ be the hidden layer state at time $\mathrm{t}$, which depends on the current input $x^{t}$ and the previous hidden layer state $H^{t-1} ; O^{t}$ be the result of the current layer at time t; $L$ be the error at time $\mathrm{t}$, which depends on the learning sample output $Y^{t}$ and the current layer output $O^{t} ; u, v$, and $w$ be the weights of the CNNRNN; $b_{1}$ and $b_{2}$ be the mutual RNN thresholds. Then, the range of hidden layer state at time $t$ can be calculated by:

$H^{t}=f\left(u \times x^{t}+w \times H^{t-1}+b 1\right)$     (4)

The expected output $0^{t}$ of the RNN at time $t$ can be calculated by:

$O^{t}=f\left(v \times H^{t}+b 2\right)$     (5)

The error L of the RNN can be calculated by:

$L=O^{t}-y^{t}$      (6)

The RNN can effectively compute crop yield, because the space variable and overfitting are restricted by two factors: the layered description of original features and the sparse constraint. Every learning instance is pre-learned before DRL. Then, the agent forecasts the yield to generate Q-range layer by layer.

DRL training involves so many states and actions that the correlations between data might be unstable. Therefore, two adjustments were made to the DQN training. The primary adjustment is understanding replay: the agent’s understanding is preserved in the replay storage (D)  using the state, action, and reward of the current timestamp, as well as the state of the next timestamp. This adjustment saves the agent’s understandings. Individual understanding $e_{t}$ at $t$ are characterized as $e_{t}=\left(e_{t}, x_{t}, a_{t}, r_{t}, s_{t+1}\right)$, and the memories time at $t$ are denoted as $D_{t}=\left\{e_{1} \ldots . . e_{t}\right\}$. Experience replay is a useful technique for eliminating parameter divergence, allowing agents to distinguish their understanding through training. The next adjustment is to allow an autonomous system to generate the goals during QL upgrading. Through the adjustments, DRL training becomes much more stabler.

Note that most RL strategies employ a Bellman equation to iteratively update the action value factor. The updating process is rather time consuming. To solve the problem, this paper predicts this factor by the weighted RNN predictor with weight $\theta$. Hence, the Q-network can be produced by decreasing the MSE in the Bellman equation by revising parameter $\theta_{i}$ in the i-th iteration.

The training process can be divided into two stages: the preprocessing of CNNRNN, and the training of DQN agent. Following the $\mathcal{E}$-greedy policy, the agent chooses and performs a random action with a probability $\mathcal{E}$. Thus, the probability of choosing the action with the highest q value equals 1-$\mathcal{E}$. This paper also adopts the stochastic gradient descent technique. Based on the training data, the optimization strategy iteratively modifies the system weights. The following is the training algorithm of the CNNRNN using DRQN.

Algorithm: Training of CNNRNNDRL

Step 1: Pre-learning the CNNRNN.

(a) Initialize the replay storage ability as N.

(b) Initialize the s-CNN, r-CNN, h-CNN, t-CNN, and a-CNN.

(c) Initialize the I number of RNN network with $\theta_{i j}$ random weights of J layers.

For i=1, I

For j=1, J do

(c) Learn the $j^{t h}$ hidden layer of $i^{t h}$ RNN

(d) Preserve the variables of the $j^{t h}$ hidden layer.

End For

End For

(e) Initialize action-value system Q using the hidden layer’s variables excluding the input and output layers.

(f) Initialize the desired action-value factor Q using similar variable as Q'.

Step 2: Learning the DQN agent.

For event $=1, M$ do

(a) Initialize the examined series $s_{1}$ via outputting the estimated yield arbitrarily.

For $t=1, \mathrm{~T}$ do

(b) Pick an arbitrary action $a_{t}$ at the probability $\varepsilon$.

(c) Execute the action and find the incentive $r_{t}$.

(d) Arbitrarily create the consecutive state $s_{(t+1)}$

(e) Preserve the storage $D$ as $\left(s_{t}, a_{t} r_{t}, s_{(t+1)}\right)$.

(f) Concerning the system variable $\theta$, execute gradient descent on:

$\left(r_{t}-Q\left(s_{t}, a_{t} ; \theta\right)^{2}\right.$

(g) Reassign $Q^{\prime}=Q$.

End For

During the construction of learning models, our method explains several factors that affect rubber yield, and assess different plant metrics. If the time series is too long, the DRL-based CNNRNN may suffer vanishing or exploding gradients.

To improve the time series for rubber plant identification, this paper builds a novel deterministic formula of probabilistic MPC with trained GP models, and ambiguity dissemination for long-term forecast. Then, Pontryagin's Maximum Principle (PMP) can be applied to the open-loop planning stage of probabilistic MPC with GPs. The PMP offers a set of principles to handle control restrictions, while retaining the necessary criteria for optimality.

The above technique theoretically justifies the detection of DRL-based rubber yield, and realizes traditional information competence in DQN, without sacrificing the excellence of probabilistic modeling. In this way, it is possible to address state and control restrictions, and maximize information competence.

3.7 Probabilistic MPC-based controller learning

Suppose there is a dynamical system with a stochastic component [14] with states $x \in R^{D}$ and admissible controls (actions) $\in U \subset R^{U}$, The states follow Markovian dynamics:

$x_{t+1}=f\left(x_{t}, u_{t}\right)+w$      (7)

With a (known) transition function $f$ and an interaural intensity difference (IID) system noise $w \sim N(0, Q)$, where $Q=\operatorname{diag}\left(\sigma_{1}^{2}, \ldots, \sigma_{D}^{2}\right)$, an RL setting is given to find the control signals $u_{0}^{*}, \ldots, u_{T-1}^{*}$ that reduce the expected longterm cost projections:

$J=E\left[\Phi\left(x_{T}\right)\right]+\sum_{t=0}^{T-1} E\left[l\left(x_{t}, u_{t}\right)\right]$,      (8)

where, $\left(x_{T}\right)$ is a terminal cost; $\left(x_{t}, u_{t}\right)$ is the cost of applying control $u_{t}$ in state $x_{t}$. It is assumed that the initial condition has a Gaussian distribution, with $p\left(x_{0}\right)=\mathcal{N}\left(\mu_{0}, \Sigma_{0}\right)$. The overall flow of the proposed method is illustrated in Figure 3.

The data efficiency is ensured by a model-based $\mathrm{RL}$ technique, in which a framework of the indefinite shift operation $f$ is created to generate open-loop $^{1}$ optimum controllers $u_{0}^{*}, \ldots, u_{T-1}^{*}$ that minimize the value of formula (7). In this way, the learnt framework is modified with the latest understanding, and re-planned after each application of the control series.

3.png

Figure 3. Overall flow of CNNRNNDRL

3.8 Long-term estimations

For a specific control series $u_{0}, \ldots, u_{T-1}$, the state distributions $p\left(x_{1}\right), \ldots, p\left(x_{T}\right)$ can be iteratively predicted by:

$p\left(x_{t+1} \mid u_{t}\right)=\iint p\left(x_{t+1} \mid x_{t}, u_{t}\right) p\left(x_{t}\right) p(f) d f d_{x_{t}}$      (9)

For $\mathrm{t}=0, T-1$, a deterministic Gaussian approximation can be made to $p\left(x_{t+1} \mid u_{t}\right)$ through moment matching.

3.9 Expected long-term cost

Adding the estimated immediate expenses to formula (8), the expected long-term cost can be estimated by:

$E\left[l\left(x_{t}, u_{t}\right)\right]=\int l\left(\widetilde{x_{t}}\right) N\left(\widetilde{x_{t}} \mid \widetilde{\mu_{t}}, \widetilde{\Sigma_{t}}\right) \widetilde{d_{x_{t}}}$      (10)

For $t=0, \ldots, T-1, l$ is chosen such that this expectation and the partial derivatives $\partial E\left[\left(x_{t}, u_{t}\right)\right] /$ $\partial x_{t}, \partial E\left[l\left(x_{t}, u_{t}\right)\right] / \partial u_{t}$ can be computed analytically.

$l_{M M}\left(z_{t}, \mathrm{u}_{t}\right)=\mathrm{l}_{\mathrm{MM}}\left(\widetilde{\mathrm{z}_{\mathrm{t}}}\right):=\mathrm{E}\left[\mathrm{l}\left(\mathrm{x}_{\mathrm{t}}, \mathrm{u}_{\mathrm{t}}\right)\right]$      (11)

$\Phi_{\mathrm{MM}}\left(\mathrm{z}_{\mathrm{T}}\right):=\mathrm{E}\left[\Phi\left(\mathrm{x}_{\mathrm{T}}\right)\right]$     (12)

The deterministic mapping (10) projects the mean and covariance of $\widetilde{x}$ into the predicted costs in formula (7). After the CNNRNNDRL is combined with MPC, our method can be denoted by CNNRNNPCDRL, which can forecast rubber yield efficiently with a shorter time series and better classification accuracy.