Vision-Based Steering Control of an Ackermann Mobile Robot on Curved Roads Using Model Predictive Control

This research focuses on the development of control systems for robots. Mobile that uses the Ackermann steering mechanism with a black line field that has 3 types of turns, namely curved with an angle of 110° to 130°, elbow with an angle of 90°, and sharp with an angle of 50° to 70°. The robot control system uses MPC; this system aims to increase the accuracy of controlling the robot's direction on the track.

2.1 Model Predictive Control

MPC is an optimal control method used to control dynamic systems by predicting the future response of the system and calculating optimal control actions based on these predictions. MPC takes into account predictions of future system dynamics and uses a system model to calculate controls that can minimize the cost function [16].

MPC is very effective for applications involving physical constraints and system limitations, and can optimize performance under dynamic and uncertain conditions. In the context of autonomous robots with Ackermann steering, MPC is used to predict and control the robot's steering angle to maintain a desired path with high precision [17, 18]. The steps for implementing MPC on a mobile robot with Ackermann steering are as follows:

a. Cost Function MPC

MPC works by minimizing a cost function that includes two main components: position error and steering angle control. Function commonly used in MPC in Eq. (1).

$J=\sum_{i=0}^{n-1}\left(\left\|x(k+i \mid k)-x_{r e f}(k+i)\right\|_Q^2+\|u(k+i)\|_R^2\right.$                (1)

where:

J: cost function,

N: prediction horizon,

$\mathrm{x}(\mathrm{k}+\mathrm{i} \mid \mathrm{k})$: Predict the state at step $\mathrm{k}+\mathrm{i}$ based on information step k,

$\mathrm{x}_{\text {ref }}(\mathrm{k}+\mathrm{i})=$ reference value for the state at step $\mathrm{k}+\mathrm{i}$,

$\mathrm{u}(\mathrm{k}+\mathrm{i})$: control prediction at step $\mathrm{k}+\mathrm{i}$,

Q: output cost function error weight,

R: input cost function weight.

The objective function in MPC is to minimize the error between the predicted vehicle position and orientation and the desired (reference) destination, as well as the control penalty $\delta(k)$. The objective function $J$ can be written as Eq. (2).

$\begin{aligned} J=\sum_{i=0}^{n-1}[(x(k+i)- & \left.x_{\text {ref }}\right)^T Q\left(x(k+i)-x_{\text {ref }}\right) \\ & +\left(y(k+i)-y_{\text {ref }}\right)^T Q\left(y(k+i)-y_{\text {ref }}\right) \\ & \left.+\left(\theta(k+i)-\theta_{\text {ref }}\right)^T Q\left(\theta(k+i)-\theta_{\text {ref }}\right)\right]\end{aligned}$                  (2)

where, $\mathrm{x}(\mathrm{k}+\mathrm{i}), \mathrm{y}(\mathrm{k}+\mathrm{i}), \theta(\mathrm{k}+\mathrm{i})=$ predicted state at step $k+ i \mathrm{x}_{\text {ref}}, \mathrm{y}_{\text {ref}}, \theta_{\text {ref}}=$ condition reference.

b. System Constraints

MPC also takes into account the physical constraints of the system, such as the maximum steering angle (δ max) and the robot's speed (vmax) to prevent the robot from straying from the desired path. These constraints are incorporated into the MPC equations to ensure that the resulting solution does not violate the robot's physical constraints.

c. Prediction and Control

The MPC predicts the system several steps into the future (time horizon) and calculates the optimal control based on the robot's dynamic model. The calculated control is the steering angle δ that will be applied to the robot to reach the desired position. At each time step, the MPC updates the prediction and calculates the optimal steering angle based on the robot's current state and detected path conditions.

d. Implementation on Robot

After the MPC calculates the optimal steering angle, it sends this command to the servo motors controlling the robot's front wheels. This process is repeated at regular intervals to adapt the robot to changes in the path or changing environmental conditions.

e. Testing and Evaluation

After the MPC implementation, testing was conducted to evaluate the system's performance in accurately following paths. The evaluation was based on parameters such as the root mean square error (RMSE) between the robot's path and the reference path. Steering angle error to the ideal direction, and response time to changes in direction on a curved track.

2.1.1 Discrete prediction model and state definition

In this study, the Ackermann vehicle is approximated using a kinematic bicycle model with wheelbase L. The robot is controlled by the steering angle δ (rad) at a constant forward speed v (m/s). The states used in MPC are the lateral tracking error e_y and the heading error e_ψ with respect to the detected trajectory centerline.

The continuous-time error dynamics are written as:

$\dot{\mathrm{e}} \_\mathrm{y}=\mathrm{v} \cdot \sin \left(\mathrm{e} \_\psi\right)$

$\dot{\mathrm{e}} \_\psi=(\mathrm{v} / \mathrm{L}) \cdot \tan (\delta)$− ψ̇_ref

where, $\psi \_$ref is the desired heading (trajectory direction) and ψ̇_ref is its rate of change. In this work, $\psi_{-}$ref is obtained directly from the vision-based road angle and ψ̇_ref is assumed to be small within one sampling period.

By discretizing with sampling time T_s using forward Euler, the prediction model becomes:

$\mathrm{e} \_\mathrm{y}(\mathrm{k}+1)=\mathrm{e} \_\mathrm{y}(\mathrm{k})+\mathrm{v} \cdot \mathrm{~T} \_\mathrm{s} \cdot \sin \left(\mathrm{e} \_\psi(\mathrm{k})\right)$

$\mathrm{e} \_\psi(\mathrm{k}+1)=\mathrm{e} \_\psi(\mathrm{k})+\left(\mathrm{v} \cdot \mathrm{T} \_\mathrm{s} / \mathrm{L}\right) \cdot \tan (\delta(\mathrm{k}))$

This discrete model is used to predict the future error states over the prediction horizon N.

2.1.2 Vision-to-state mapping (canny–hough output interface)

Trajectory detection provides two outputs every frame: (1) road angle θ_road (deg) from the dominant Hough line, and (2) lateral deviation d_px (px) between the image center and the detected trajectory centerline. The image coordinate is defined with x-axis horizontal (positive to the right) and y-axis vertical (positive downward).

The desired heading ψ_ref is defined as ψ_ref = θ_road (converted to radians). The robot heading ψ is assumed aligned with the camera optical axis. Therefore, the heading error is approximated as:

e_ $\psi \approx-\psi \_$ref

The lateral error is obtained from d_px by a pixel-to-meter scaling factor $\mathrm{s}(\mathrm{m} / \mathrm{px})$:

$\mathrm{e} \_\mathrm{y}=\mathrm{s} \cdot \mathrm{d} \_\mathrm{px}$

The scaling factor s is determined from the known track width in the image at the Region of Interest (ROI) and is kept constant during experiments. This mapping allows the vision module to provide the MPC with physically meaningful state variables.

2.1.3 MPC formulation, parameters, and constraints

At each sampling instant k, the MPC solves an optimization problem to minimize the tracking error and control effort:

$\begin{gathered}\min \_\{\delta(\cdot)\} \mathrm{J}=\Sigma \_\{\mathrm{i}=1 . . \mathrm{N}\}\left(\hat{\mathrm{x}}(\mathrm{k}+\mathrm{i} \mid \mathrm{k})-\mathrm{x} \_ \text {ref }\right)^{\mathrm{T}} \mathrm{Q}(\hat{\mathrm{x}}(\mathrm{k}+\mathrm{i} \mid \mathrm{k}) \\ \left.-\mathrm{x}_{\_} \text {ref }\right)+\Sigma \_\{\mathrm{i}=0 . . \mathrm{M}-1\} \delta(\mathrm{k}+\mathrm{i} \mid \mathrm{k})^{\mathrm{T}} \mathrm{R} \delta(\mathrm{k}+\mathrm{i} \mid \mathrm{k})\end{gathered}$

with $\mathrm{x}=\left[\mathrm{e} \_\mathrm{y}, \mathrm{e} \_\psi\right]^{\mathrm{T}}$ and $\mathrm{x} \_$ref $=[0,0]^{\mathrm{T}}$. The control sequence is constrained by the steering limits:

$\delta \_$min $\leq \delta(\mathrm{k}+\mathrm{i} \mid \mathrm{k}) \leq \delta \_$max

In the implemented prototype, $\delta$ is limited to $\pm 40^{\circ}$ to prevent mechanical saturation of the steering linkage. The MPC is executed with the following parameters: sampling time $\mathrm{T} \_\mathrm{s}=$ 0.10 s , prediction horizon $\mathrm{N}=10$, control horizon $\mathrm{M}=3, \mathrm{Q}= \operatorname{diag}([1.0,0.5])$, and $\mathrm{R}=0.1$. A quadratic programming (QP) solver is used to compute the optimal $\delta$ sequence; only the first control input $\delta(\mathrm{k} \mid \mathrm{k})$ is applied to the servo (receding horizon principle).

2.1.4 Baseline controllers and evaluation metrics

To objectively evaluate the benefits of MPC, two baseline controllers are considered: (i) a proportional steering controller (P-controller) that maps e_ψ and e_y into δ, and (ii) a Pure Pursuit controller using the detected lane centerline as a reference. All controllers are tested on the same track scenarios and at the same constant speed.

Tracking performance is evaluated from logged data over a full lap using Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and maximum deviation:

$\begin{gathered}\operatorname{MAE}=(1 / T) \Sigma\left|\mathrm{e} \_\mathrm{y}(\mathrm{t})\right|, \operatorname{RMSE}=\operatorname{sqrt}\left((1 / T) \Sigma \mathrm{e} \_\mathrm{y}(\mathrm{t})^{\wedge} 2\right), \\ \text { e_y,max }=\max \left|\mathrm{e} \_\mathrm{y}(\mathrm{t})\right|\end{gathered}$

Control smoothness is evaluated by the steering rate Δδ = δ(k) − δ(k−1) and its standard deviation. These metrics are reported in the Results section to support quantitative comparison.

2.2 Canny Edge Detection

Canny Edge Detection aims to identify significant changes in image intensity that indicate the boundaries or edges of objects in the image. These edges are areas where large changes in pixel intensity occur, often marking the boundary between an object and its background. The theory behind this method is that these edges provide important information about the structure and shape of objects in the image [19-27]. Canny Edge Detection works through a series of steps designed to provide smooth and accurate edge detection. Here are the main steps involved:

1. Refinement

The first step in the canny algorithm is image smoothing to reduce any noise that may be present in the image, which could interfere with edge detection. To this end, a Gaussian filter is used to smooth or smooth the image. Gaussian smoothing works by giving more weight to pixels closer to the center and decreasing it towards the edges, resulting in a smoothing effect that effectively reduces noise. The smoothing effect is calculated using Eq. (3).

$G(x, y)=\frac{1}{2 \pi \sigma^2} e^{-\frac{x^2+y^2}{2 \sigma^2}}$                  (3)

Information:

1. G(x,y): Gaussian value at coordinates (x,y).
2. σ: Parameter that controls the level of image smoothing.

2. Gradient Calculation

After the image is smoothed, the next step is to calculate the image intensity gradient at each pixel. This gradient describes the rate of change in image intensity, which indicates the presence of edges. To calculate the gradient, the Sobel operator is used, which applies convolution to the image to obtain the gradient components in the horizontal (G_x) and vertical (G_y) directions. The gradient is calculated by Eq. (4).

$\mathrm{G}_{\mathrm{x}}=\frac{\partial \mathrm{I}}{\partial_{\mathrm{x}}}, \mathrm{G}_{\mathrm{x}}=\frac{\partial \mathrm{I}}{\partial_{\mathrm{y}}}$             (4)

Information:

1. I = Input image.
2. G_x = image gradient in horizontal direction.
3. G_y = image gradient in the vertical direction.

Total gradient G is calculated using Eq. (5).

$G=\sqrt{G_x^2+G_y^2}$             (5)

where, G is the gradient magnitude, which indicates the magnitude of the intensity change at each point. The direction of the gradient θ is calculated by Eq. (6). The direction of this gradient provides information about the orientation of the detected edge.

$\theta=\operatorname{atan} 2\left(G_y, G_x\right)$              (6)

3. Non - Maximum Suppression

This process is done by comparing the gradient value at each pixel with the gradient values at the surrounding pixels along the gradient direction θ. If the gradient value at a pixel is not greater than the gradient value at its neighboring pixel, then the pixel is considered not to be part of the edge and is removed.

4. Thresholding with Hysteresis

Pixels with gradient values greater than the high threshold are considered strong edges. Pixels with gradient values smaller than the low threshold are considered non-edges. Pixels with gradient values between these two thresholds are considered weak edges, which are only retained if they are connected to a strong edge.

5. Edge Tracking by Hysteresis

Pixels that are considered weak edges and are connected to strong edges are retained. This process ensures that only significant and relevant edges are retained, while edges that are not connected to strong edges are removed. This process helps reduce interference or noise that can be introduced into edge detection.

2.3 Hough Transform

Hough Transform is a technique used in image processing to detect certain geometric shapes in images, especially to detect straight lines. This technique was introduced by Paul Hough in 1962 and is used to detect objects in images that are not explicitly defined in image space. One of the main applications of the Hough Transform is in the recognition of lines or other geometric shapes, such as circles, in images. The Hough Transform converts the usual image space (Cartesian space) into a parameter space, where each point in the image is represented as geometric parameters (such as angles and line lengths), leading to a representation that is easier to process [28-30]. In 2D image processing, the Hough Transform focuses on line detection, which can be described by Eq. (7).

$\rho=x \cos (\theta)+y \sin (\theta)$               (7)

Information:

1. ρ is the shortest distance from the point to the origin of coordinates (the detected line),
2. θ is the angle between the line and the x-axis,
3. x and y are the coordinates of a point in image space.

A line in image space cannot be represented by a single coordinate pair (x, y) because a line is a two-dimensional concept that requires two parameters to define it, namely ρ and θ. Therefore, the Hough Transform uses this parameter space to detect lines in an image by converting the image into a parameter space that is easier to analyze. The steps in the Hough Transform for Line Detection are as follows:

1. Transformation of Image Points to Parameter Space
2. Sound Collection in Parameter Space (Accumulator Array)
3. Determination of the Edges of the Most Common Lines
4. Convert Back to Image Space

2.4 Block diagram

The block diagram of this system, Figure 1 explains the workflow of the system to be designed. The system starts with a “Reference” that represents the robot’s position relative to the desired path in units of px, then the “MPC Controller” calculates the control signal u_n given to the “Plant” (mobile robot). The steering angle δ_n is obtained and produces the output of the robot’s x and y positions relative to the path. Then a check is carried out by the “Camera” which becomes feedback to measure the actual position of the vehicle, and the results are used by the MPC to adjust the control signal so that the vehicle remains at the desired Reference, namely the 0 px angle.

Figure 1. System block diagram

This mobile robot is equipped with an Ackermann mechanism steering controlled by MPC based on the results of trajectory detection using a camera and the Canny Edge Detection algorithm. This system was built with an integrated approach between a computer, an ESP32 microcontroller, and drive and steering actuators, as shown in Figure 2.

Figure 2. Input process output

2.5 Flowchart

In designing this system, there are several flowcharts as algorithms for how the system will work, as explained below:

1. System Flowchart

Explained in the system flowchart Figure 3, there is a declaration of variables to be used consisting of "Trajectory Image = CL", "Road Angle = SJ", and "Steering Angle = δ ". After the variable declaration, there is "INPUT (CL)", where the camera takes real-time recordings. The trajectory image that has been detected by the camera is immediately processed in "Line Detection", where this process will search for road boundaries and update the angle of change of the road. After the angle of change of the road is obtained, the angle of the road will be processed by "MPC" to predict how many degrees the steering angle will change so that the mobile robot does not leave the track. Then we get "OUTPUT (δ)" which will direct the robot to stay on its path, and the system is complete.

3.png

Figure 3. System flowchart

2. Line Detection Flowchart

The system in digital image processing, where the camera detects the road and is processed in the line detection flowchart Figure 4. The system starts from the declaration of the variables "Edge Image = CE", and "Road Angle = SJ". Followed by "Image Pre-processing" where the image of the path will be improved image quality and remove noise in the image. After that the image will be processed in "Canny Edge Detection", where the road image will be taken only the edge of the road to determine the boundaries and shape of the path. After the edge of the road is known, it will be processed again using the "Hough Transform", in this process, the image will be drawn a straight line to determine the change in the angle of the road. After both are obtained, namely the edge boundary and the angle of the road, the results will be filtered and combined. To reduce the occurrence of unwanted detection, it is continued with conditioning, namely, whether the edge is detected. If the edge of the path is not detected, the system is restarted from "INPUT (CL)". And if the edge of the path is detected, then "OUTPUT (SJ)" is obtained, and the line detection process is complete.

4.png

Figure 4. Flowchart line detection

3. Canny Edge Detection Flowchart

Figure 5 shows the flowchart of Canny edge detection, starting with the declaration of the variable “Citra Edge = CE”, followed by image smoothing using a Gaussian filter to reduce noise. Next, “Computing Gradient” is used to obtain the edge direction. After that, the non-maximum Suppression is performed to preserve edge peaks, followed by “Double Threshold” to identify strong and weak edges. Weak edges connected to strong edges are preserved. The result is a clear edge image, and the process is complete.

5.png

Figure 5. Flowchart edge detection

4. Hough Transformation Flowchart

The Hough Transform is used to detect geometric shapes, such as straight lines, in the image of Figure 6. The process begins with the declaration of the variable “Citra Edge = CE” and continues with “INPUT (CE)”, then transforms the points in the image into a parameter space, where each line in the image is represented by two parameters: the line length (r) and the angle (θ), which describes the orientation of the line. Next, accumulation is carried out using an accumulator array to calculate the contribution of each point to these parameters, which are used to determine the existing lines. The lines with the highest contribution are considered as detected lines. The detected line information is then returned to the image space to be drawn, and the process is complete.

6.png

Figure 6. Hough Transform flowchart

5. Flowchart MPC

Figure 7 shows the MPC flowchart, the system starts from the declaration of the variables “Road Angle = SJ”, and “Steering Angle = δ”, then there is “INPUT (SJ)” and calculates the objective function that measures how far the current system condition is from the desired reference condition. The objective function is then optimized to determine the optimal control, and then the “OUTPUT (δ)” is obtained, which will later be used by the servo motor as the Ackermann angle driver. Steering and the process are complete.

Figure 7. Flowchart Model Predictive Control (MPC)

2.6 Tool design

The mobile robot design in this study was made to resemble a four-wheeled vehicle in general, which uses an Ackermann steering mechanism. Steering. This design was chosen because it fits the characteristics of the robot, which will be designed to maneuver stably on curved paths. The robot's main structure uses acrylic, chosen for its sufficient strength to support all components.

The robot has four wheels, with the two front wheels serving as steering wheels and the two rear wheels serving as driving wheels. The Ackermann steering mechanism is controlled using a single servo motor connected to the two front wheels via a linkage system as shown in Figure 8. For forward movement, a single DC motor mounted on the axle is used. The gears transmit motor power to the rear wheels. This system allows the robot to perform turning maneuvers at angles adjusted automatically through MPC.

Figure 8. Robot component parts

The overall dimensions of the robot are 25 cm long, 15 cm wide, and 17 cm high, which are adjusted to be sufficient for testing in an indoor environment while still representing a small four-wheeled vehicle. The design of the mobile robot is shown in Figure 9.

Figure 9. Robot dimensions

Visually, the robot's view direction consists of a front view in Figure 10. You can see the camera facing forward and the servo drive, half of the body of which is under the car, and 2 LEDs located on the front of the robot to light the way if needed.

Figure 10. Front view

Rear view of Figure 11 shows 1 yellow button to set the robot's condition to be ready or not ready, and 1 red switch, which will be used as a switch. Supply from the battery.

Figure 11. Rear view

The side view in Figure 12 shows two wheels, namely the front wheel and the rear wheel, which are 15 cm apart. The camera orientation is slightly tilted downwards.

Figure 12. Side view

2.7 Road terrain

The terrain to be used is printed on . The road width is 20 cm. This has 2 colors, consisting of black as the main road area and white as the . This has several types of turns consisting of curved turns with an angle of 110° to 130°, T-turns with an angle of 90, and sharp turns with an angle of 30° to 50°. This design was created in as shown in Figure 13.

Figure 13. Road terrain

2.8 Electronic circuits

Electronic circuits on mobile, this robot is designed to connect all the main components so that the system can work properly. The ESP32 microcontroller acts as a control center that receives angle data from the computer via a WiFi network, then controls the robot's movement based on that data. The servo motor is connected to the ESP32 and is used to control the turning angle of the front wheels (Ackermann steering). A DC motor at the rear is used to move the robot forward. The DC motor is controlled through a motor driver, which is also connected to the ESP32. The electronic circuit is made using Fritzing software, as shown in Figure 14.

Figure 14. Electronic circuits

2.9 Ackermann steering kinematic model

The kinematic model of the robot in Figure 15 is used to predict the future position and orientation of the robot. The kinematic model of the robot is written in the form of equations of motion that depend on the speed and steering angle of the robot in Eq. (8).

$\operatorname{Tan}(\delta)=\frac{\mathrm{L}}{\mathrm{R}}$             (8)

Information:

1. δ: turning angle,
2. L: distance between front axles and rear (wheelbase),
3. W: wheelbase width (track) width),
4. R: turning radius from the center of the turn to the center point of the vehicle.
5. ICR: Instantaneous Center of Rotation

Figure 15. Ackermann steering kinematic model [31-34]