QoS-Aware Task Offloading in Fog-IoT Environment Using Hybrid Deep Q-Learning with Actor-Critic Framework

The incorporation of IoT technology into applications will require computational infrastructures that satisfy demands for low latency, low-cost scaling, and energy efficiency. With communication delays and bandwidth constraints, conventional cloud computing models are typically unable to satisfy those demands. As a way around these drawbacks, a distributed approach termed fog computing has been proposed that shifts its computational and storage infrastructure to within servile distance from IoT devices. This facilitates efficient load offloading and improved QoS provisioning.

Recent works have explored optimization, and reinforcement learning techniques for task allocation approaches for load balancing in Fog-IoT computing architectures. For example, Gupta and Singh [1] developed Fog-GMFA-DRL, a hybrid model using Grey Wolf and Modified Moth Flame Optimization with deep reinforcement learning (DRL) for load balancing in Fog-IoT architectures. Joshi and Srivastava [2] proposed a double-auction based scheduling mechanism that schedules tasks for three-tier cloud–fog–IoT systems that adheres to QoS guarantees while ensuring an equitable level of tasks, scheduling, and specifications.

Chalapathi et al. [3] provided a detailed survey of the role of fog and edge computing in improving IIoT applications by reducing latency, improving scalability, and supporting real-time analytics. The review also discusses potential research challenges and opportunities for the future towards secure, energy-efficient, and privacy-preserving IIoT deployments. Peng et al. [4] also provided a thorough review of computation offloading via DRL, and its adaptability to dynamic and heterogeneous contexts. Pakmehr [5] further stressed future research opportunities on offloading through DRL-based systems, particularly in relation to issues of security and efficiency. Various other works have focused on algorithmic optimization and reinforcement learning-based offloading methodologies. Allaoui et al. [6] reviewed reinforcement learning algorithms for offloading IoT tasks in fog systems, while Mashal and Rezvani [7] implemented DRL based multi-objective optimization, in which they paired trade-offs with latency, energy consumption, and cost. Wang et al. [8] followed this trajectory with a review of IoT task offloading in multi-access edge computing MEC, emphasizing the importance of scalability and energy efficiency for practical implementations. More recent works have mentioned rolling out dynamic frameworks, as well as an advanced signal DRL-based system. Abdelghany [9] demonstrated an overall improvement in flexibility and scalability of a dynamic resource management and task offloading framework for fog computing. Likewise, Tomar et al. [10] presented a DRL-based offloading mechanism for IoT devices connected to fog-enabled environments, demonstrating significant improvements in execution flexibility and reduction in latency.

Although this research demonstrates notable progress, important issues remain unaddressed. Most of the existing approaches either consider task offloading as a single-objective optimization or face scalability and convergence problems in large-scale, dynamic Fog-IoT environments. In this paper, we present a new QoS-aware hybrid Deep Q–learning method for task offloading in Fog-IoT settings that overcomes these barriers by coupling adaptive optimization with DRL to co-optimize latency, energy, and utilization simultaneously.

The major contributions of this study are:

1. A H-DQAC method for adapting and efficient task offloading in Fog-IoT settings reliably.
2. Coupling adaptive problem optimization strategies with DRL to promote faster convergence and scalability.
3. Benchmarking the proposed method against relevant state-of-the-art methods to demonstrate enhancements in latency, energy, and system throughput in Fog-IoT settings.

The rest of the paper is organized as follows: Section II discusses previous research conducted by many researchers. In Section III, the hybrid designer proposes the Deep Q-Learning with Actor-Critic (HDQL-AC) framework describing the system model, formulation of the mathematical model, and objectives for multi-objective research in terms of latency, energy, and reliability. Section IV includes explanations for the set-up for computational experiments, simulation scenarios, and performance metrics, followed by a comprehensive results and discussion section comparing the proposed model to baseline designs on key measures such as latency, effective energy consumption, the successful task ratio, throughput, bandwidth utilization and overall utility of QoS performance. Section IV provides a comprehensive analysis to identify the important findings and novel contributions of this work, while also highlighting their significant and wide-ranging implications for guiding future research on the topic of QoS-aware task offloading in dynamic Fog-IoT networks.

Existing task offloading frameworks within fog and vehicular computing demonstrate improvements in latency reduction, energy efficiency, and resource utilization. However, most of these approaches are disadvantaged by high convergence time, limited adaptability to the dynamic nature of vehicular mobility, ineffectiveness of task priority-awareness, and synchronization overhead when working within a federated learning setting. To address these shortcomings, this research presents a H-DQAC based task offloading and scheduling framework that integrates mobility-awareness, task priority constraints, and dynamic resource adaptation with scalability across distributed fog nodes. Figure 1 shows the design diagram of the proposed model.

1.png

Figure 1. Proposed model design diagram

3.1 System architecture

The proposed framework has three main layers:

• IoT/Vehicle Layer: End devices vehicles, sensors, IoT devices generate computational tasks that vary in size, deadline, and energy requirements. Each task occupies the tuple $T_i=\left\{s_i, d_i, p_i, e_i\right\}$, which consists of size $s_i$, deadline $d_i$, priority $p_i$, and required energy $e_i$.
• Fog Layer: Several fog nodes produce low-latency processing, but fog servers executing offloaded tasks perform limited and resource-constrained processing. Fog servers also collaborate via federated learning and maintain their own localized DRL agents.
• Cloud Layer: Provides backup computation/processing and has a global coordination role; however, it is only utilized when there are inadequate fog resources available.

A federated controller updates the distributed fog nodes with justification and occurs simultaneously to learn an overall representative DRL model, while data is never shared in raw form. This immensely increases data privacy and minimizes bandwidth consumption. The optimization of task offloading can be formulated as a multi-objective minimization objective. $\min \left\{L_{\text {avg}}, E_{\text {tot}}, C_{\text {tot}}\right\}$ where, $L_{\text {avg}}$ is the average latency across all tasks, $E_{\text {tot}}$ is the total energy consumed, and $C_{\text {tot}}$ is the total cost that accounts for resource utilization and communication overhead. Constraints include; deadline constraint: $L_i \leq d_i$, for each task $T_i$, resource constraint: $\Sigma_{R_j \leq R} f o g$, where R_j is allocated resource and $R_{f \circ g}$ is the fog node's capability. And priority constraint: High-priority tasks are allocated first $p_i>p_j \Rightarrow L_i^{\text {fitted}}$ with a local DRL agent. The task offloading decision is modelled as a Markov Decision Process MDP:

• State S: Encodes queue length, available bandwidth, CPU cycles, task size, deadline, and mobility context.
• Action A: Decide whether to: i execute locally, or ii offload to a neighboring fog node, or iii offload to the cloud.
• Reward: Designed to minimize delay and energy while maximizing task success ratio:

The agents learn from Deep Q-Networks DQN or Actor-Critic policies, and local models are updated periodically through federated averaging FedAvg, allowing for global learning with no raw task data being communicated.

• Task Generation: IoT devices/vehicles generate tasks with parameters.
• Local Decision Making: The fog node's DRL agent makes a decision to either execute the task locally or offload it to the cloud for executing the tasks.
• Federated Learning: Local agents learn to train the DRL agents leveraging local experiences, and they will provide the global server with model parameters at periodic intervals.
• Global Aggregation: The federated controller collects updates using FedAvg and provides the improved global policy back to the agents.
• Adaptiveness of Scheduling: During offloading tasks, the global model will adapt the decision within to minimize the latency, energy, and cost while taking into consideration the task's priority and mobility patterns.

Here presenting the proposed H-DQAC model. Eq. (1) indicates the time it takes to transmit a task from a device to a fog node, which is dependent on the task size, the bandwidth, and the propagation delay. This is indicative of both data transfer and network distance. A greater bandwidth would result in less delay, whereas a greater propagation distance results in latency.

$T_{i, j, k}^{t x}(t)=\frac{s_{i, k}}{b_{(i, j)}(t)}+\tau_{i, j}^{\text {prop}}$                        (1)

This models the time for the uplink to transmit task k from device $i$ to fog node $j$. Here, the variable $s_{i, k}$ refers to the amount of input data from the task. The variable $b_{(i, j)}$ represents the available uplink bandwidth. The $\tau_{i, j}^{\text {vrop }}$ variables to the propagation delay. As such, the time for uplink resultant from data transmission plus the network distance. Eq. (2) is used to determine processing time as a function of the task CPU demand capacity of the node. Faster nodes significantly decrease the execution delay. It helps select whether to execute a task locally, fog, or cloud.

$T_{i, k, x}^{p r o c}(t)=\frac{c_{i, k}}{f_{x(t)}}$                       (2)

The processing time for the kth task is proportional to the required CPU cycles $c_{i, k}$ of task $k$, and inversely proportional to some node processing capacity $f_x$ of node $x$, where for any $\operatorname{local}_x=i, f o g_x=j$, or $\operatorname{cloud}_x=C$. Eq. (3) estimates the average waiting time at fog nodes based on the M/M/1 queue model. As the system utilization nears full capacity, the average waiting time increases rapidly. It also reflects the effects of congestion in a busy fog node.

$W_j(t)=\frac{\rho_j(t)}{\mu_j(t)\left(1-\rho_{j(t)}\right)}, \rho_j(t)=\frac{\lambda_{j(t)}}{\mu j(t)}$                      (3)

The M/M/1 model captures the average queue time process tasks wait at fog node j (where $\lambda_j$ is the task arrival rate, $\mu_j$ is the service rate, and $\rho_j$ is the utilization of the system). When $\rho_j$ higher utilization causes a sudden spike in queuing delay. Eq. (4) expresses total latency $\left(L_j\right)$ when a task is offloaded on a fog node, combining transmission time to fog, queuing time in fog, processing time, and return time ($T^{\text {ret}}$). Since the complete latency measure incorporates all elements, it captures in total how any real-time QoS requirements met.

$\begin{aligned} L_{(i, k)}(t ; a i, k=j) & =T_{i, j, k}^{(t x)}(t)+W_j(t)+T_{i, k, j}^{\text {proc}}(t)+T_{i, j, k}^{\text {ret}}(t)\end{aligned}$                   (4)

This is the complete latency model for offloading task $k$ to fog node $j$, which includes the same components of tasks' transmission time to fog, queuing time in fog, processing time in fog, and retum time ($T^{\text {ret}}$), which we typically consider trivial. Eq. (5) formulates the overall latency during local execution, incorporating waiting time in device ($w_i$) and the local execution processing time ($t_i^{\text {proc}}$). Local execution eliminates the transmission time, but processing time will be slower if device resource is constrained. Balancing the latency aspect contributes to how we view a task's local vs. fog offloading decision.

$L_{i, k}\left(t ; a_{i, k}=0\right)=W_i^{l o c}(t)+T_{i, k, j}^{p r o c}(t)$                      (5)

Here, $L_i^{\text {local}}$ is Latency under local conditions including wait time and local processing time. Eq. (6) captures the energy cost ($\left.E_i^{\text {local}}\right)$ associated, where the energy cost of local execution will be quadratic in its CPU cycle frequency. The faster the execution time, the more energy cost there will be per task. Thus, devices with limited battery capacity must balance the performance of quick executing local tasks expending more battery energy vs. offloading which did not expend the battery's energy.

$E_{i, k}^{\text {comp }}=\kappa c_{i, k}\left(f_i^{l o c}\right)^2$                        (6)

Here, $E_i^{\text {local}}$ is the local energy, kappa is the device capacity coefficient, $f_{\text {dev}}$ is the device CPU rate, and $c_i$ is the CPU cycles. Eq. (7) shows that the transmission energy depends on the transmit power of the device and the time it takes to send the data. The larger the tasks or the weaker the bandwidth, the greater will be the energy cost of the transmission. This represents the overhead associated with offloading tasks.

$E_{i, j, k}^{t x}=P_i^{t x} T_{i, j, k}^{t x}(t)$                        (7)

Here, $E_i^{t x}$ is the transmission energy, $P_{t x}$ is the transmit power, and $t_i^{t x}$ is the transmission time. This accumulates the energy consumed locally, in fog, and in the cloud, for both computation and communication costs. Eg. (8) provides the total energy footprint of the system.

$E_{i, k}\left(t ; a_{i, k}\right)=\left\{\begin{array}{lr}E_{i, k}^{\text {comp}}, & a_{i, k}=0 \\ E_{i, j, k}^{t x}+E_{j, k}^{\text {proc}}, & a_{i, k}=j, j \in F \\ E_{i, C, k}^{t x}+E_{C, k}^{\text {proc}}, & a_{i, k}=C\end{array}\right.$                       (8)

Here, $E_{i, k}$ is the total energy, $E_{j, k}^{\text {proc}}$ is the local energy, $E_{i, C, k}^{t x}$ is the transmission energy, $E_{i, k}^{\text {comp}}$ is the fog or cloud computation energy. This binary function tests to see if the task exceeds the deadline. This is important for time-sensitive IoT applications. Each missed deadline represents a global degradation of QoS as seen in Eq. (9).

$I_{i, k}^{m i s s}=\left\{\begin{array}{lr}1 & \text {if} L_{i, k}>d_{i, k} \\ 0 & \text {otherwise}\end{array}\right.$                  (9)

Here, $L_{i, k}$ is the latency of task $i, d_{i, k}$ is the deadline of task i. Average latency, average energy consumption, and deadline miss ratio are calculated across all tasks to summarize a performance metric for the system as a whole. These metrics will be used to calculate performance metrics as shown in Eq. (10).

$\begin{gathered}\bar{L}=\frac{1}{|T|} \sum_{i, k} L_{i, k}, D M R=\frac{1}{|T|} \sum_{i, k} I_{i, k}^{m i s s}, \bar{E}=\frac{1}{|T|} \sum_{i, k} E_{i, k}\end{gathered}$                    (10)

Here, $\bar{L}$ and total energy consumption is represented by $\bar{E}$ while a deadline miss ratio is represented as DMR, and N is number of tasks used for sizing the ratio versus tasks. Eq. (11) shows the multi-objective optimization problem as a weighted sum of all the objectives including latency, energy, and reliability. The weighted terms allow for prioritizing some of the system objectives over phases while in general the objective function is driven to create task performance that achieves the multi-objective to the highest possible degree.

$J=\alpha \frac{\bar{L}}{L_{\max}}+\beta \frac{\bar{E}}{E \max}+\gamma D M R$                      (11)

Here, an objective function is represented by $J$ and the weighting factors are $\alpha, \beta$ and $\gamma$. Eq. (12) shows the reward component is meant to directly penalize high delay and energy along the deadline violations from high violations. The higher the reward the more scheduling efficiencies can be attributed to optimal DRL policies.

$r_{i, k}(t)=-\left(\alpha \frac{L_{i, k}}{L_{\max}}+\beta \frac{E_{i, k}}{E_{\max}}+\gamma I_{i, k}^{m i s s}\right)$                   (12)

Here, the reward is represented by $r_{i, k}(t)$, latency for a task is represented by $L_{i, k}$, energy consumption per task is $E_{i, k}$, total number of tasks is $I_{i, k}^{\text {miss}}$, weighting factors are $\alpha, \beta$ and $\gamma$. The state vector in Eq. (13) includes features pertaining to the system and task. These features include task size, bandwidth, CPU cycles, and mobility. A state rich in features will provide proper actions based on knowledge of the immediate surroundings.

$s_{i, k}(t)=\left[\begin{array}{c}q_i(t), q_{j 1}(t), \ldots, q_{j m}(t), \\ b_{i, j 1}(t), \ldots, b_{i, j m}(t), \frac{c_{i, k}}{f_i^{l o c}}, \frac{d_{i, k}}{T_{\max}}, \\ p_{i, k}, \text {mobility}_i(t)\end{array}\right]$                       (13)

Here, $s_{i, k}(t)$ is the state at time t, which contains the following features: $\left\{q_j, b_{i j}, c_i, d_i, p_i, v_i\right\}$. The optimal Q-value will be defined through the recursive relationship in Eq. (14). This recursive equation accounts for the immediate reward and discounted expected future rewards. This is a basis for creating policies through reinforcement learning.

$Q^*(s, a)=E\left[r+\gamma_{\text {disc}} \max _a^{\prime} Q^*\left(s^{\prime}, a^{\prime}\right) \mid s, a\right]$              (14)

Here, $Q^*(s, a)$ is an optimal Q-value, $r$ is the reward, $\gamma$ is the discount factor, and $s$ is the next state.

Eq. (15) defines the target Q-value in the standard DQL paradigm. The target Q-value also selects the maximum Q-value of the next state. This can lead to overestimation bias.

$y_t=r_t+\gamma_{\text {disc}} \max _{a^{\prime}} Q\left(s_{t+1}, a^{\prime}; \theta^{-}\right)$                       (15)

Here, $y_t$ is the target value, $r$ is the reward, $\gamma$ is the discount factor, and $Q$ is the Q-function. The variation in Eq. (16) decouples action selection from Q-value evaluation to reduce overestimation bias through the use of two distinct networks for action selection and evaluation, providing a more stable training process.

$\begin{aligned} & y_t^{D D Q}=r_t+\gamma_{\text {disc}} Q\left(s_{t+1}, \text {arg} \max _a^{\prime} Q\left(s_{t+1}, a^{\prime} ; \theta\right) ; \theta^{-}\right)\end{aligned}$                     (16)

Here, $y_t^{D D Q N}$ is the double DQN target, $a$' is the best action from the online network, and $Q()$ is the target Q -network. Eq. (17) represents the loss function which minimizes the discrepancy between predicted Q-values and target Q-values. This drives Q-network updates toward optimality; a lower loss implies more accurate action-value predictions from the Q-network.

$L(\theta)=E_{\left(s, a, r, s^{\prime}\right) \sim D}\left[\left(y_t-Q(s, a ; \theta)\right)^2\right]$                     (17)

Here, $L$ represents loss, $y_t$ represents target Q -value, and $\mathrm{Q}(\mathrm{s}, \mathrm{a}); \theta$ represents the predicted Q -value. Eq. (18) represents the actor loss function, which ensures the policy is selecting actions that have high advantages, thus maximizing the expected performance increase.

$L_{\text {actor}} \phi=-E_{s \sim D}\left[E_{a \sim \pi \phi(\mid s)}\left[A_\psi(s, a)\right]\right]$                     (18)

This promotes rapid adaptability. $L_{\text {actor}}$ is actor loss, pi is policy, and $A_\psi(s, a)$ is the advantage function. Eq. (19) represents the critic loss function, which ensures the predicted state values are aligned with the expected return. Decreasing this error stabilizes the actor's learning/update process, thus enhancing the stability of learning.

$L_{\text {critic}}(\psi)=E_{\left(s, r, s^{\prime}\right) \sim D}\left[\left(r+\gamma_{\text {disc}} V_\psi\left(s^{\prime}\right)-V_\psi(s)^2\right)\right]$                     (19)

Here, $L_{\text {critic}}$ is critic loss, $V_\psi(s)$ is the predicted value, and $y$ is the target return. Eq. (20) serves to aggregate these localized model updates into a single global model. Each contribution is weighted by the sizes of the client data sets, allowing them to collaboratively learn from their experiences while guaranteeing individual autonomy.

$\theta_g=\frac{1}{|S|} n_i \theta_i$ or normalized: $\theta_g=\frac{\sum_j n_j \theta_i}{\sum_i n_i}$                        (20)

Here, $\theta_g$ is global weights, $\theta_i$ is local weights, $n_i$ is local client data samples, and S is all samples. In Eq. (21), TD-error sampled experiences receive higher replay probability the more TD-error they have, which speeds up learning by emphasizing harder experiences.

$P(k)=\frac{\left|\delta_k\right|^\alpha}{\sum_j\left|\delta_j\right|^\alpha}$                        (21)

This increases the efficiency therefore learning will happen faster in a DRL setting, $P(k)$ is the probability from sampling, $\delta_j$ is TD error, and $\alpha$ is a priority factor. In Eq. (22), we simply extend the optimization objective with the addition of a penalty for violating deadlines. The Lagrangian form ensures a promised level of QoS will be satisfied under these constraints. The form will balance feasibility and optimality as observed in the dual objective formulation of the Lagrangian.  

$L_{\text {total }}(\theta, \lambda)=J(\theta)+\lambda(D M R(\theta)-\eta), \lambda \geq 0$                          (22)

Here, $J$ is the Lagrangian objective, $\lambda$ is the multiplier, and $\eta$ is the threshold for QoS. In Eq. (23), we outline the adaptive multiplier update3 based on whether constraints are being violated. If a constraint to a deadline is being missed and exceeded a penalty multiplier, the penalty weight will keep increasing.

$\lambda=[\lambda+\rho(\operatorname{DMR}(\theta)-\eta)]^{+}$                       (23)

This rule will help ensure reliability and sustainability within the system. Here, $\lambda$ is the multiplier, $\rho$ will be the learning rate, $D M R(\theta)$ is the violation, and $\eta$ is the threshold.

The HDQL-AC framework combines the value-based learning process of the DQN with the policy-gradient stability of the Actor−Critic. The DQN is able to estimate Q-values for initial offloading decisions, while the actor–critic is refining the action policy in order to maximize long-term QoS rewards. The actor outputs the continuous probability distributions over the tasks, while the critic evaluates them through reward feedback. The combined loss function can be seen in Eq. (24) with αbalancing value and policy gradients.

$\begin{aligned} L_{H D Q L-A C}=\alpha\left(r_t+\frac{\gamma}{\max Q\left(s_{\{t+1\}}, a^{\prime}; \theta^{-}\right)}\right)+(1-\alpha)\left(V\left(s_t; \phi_c\right)\right. \left.\left.-A\left(s_t, a_t; \phi_a\right)\right)\right]^2\end{aligned}$                (24)

Mobility is represented in Eq. (25).

$h_{i, k}(t)=h_0\left(\frac{d_0}{d_{i, k}(t)}\right)^\eta$                       (25)

Here, $d_{(i, k)(t)}$ changes based on user velocity $v_{i(t)}$ drawn from a truncated Gaussian distribution. This way, the state vector captures the impact of changing mobility on link quality and task allocation.