Deep Reinforcement Learning-Based Adaptive Discrete Phase-Shift Optimization for Intelligent Reflecting Surface Supported Wireless Communications

The rapid expansion of wireless communication systems and the increasing need for higher data rates, more coverage, and improved energy efficiency have sparked extensive study into technologies that may alter the physical propagation environment. Among these developing technologies, intelligent reflecting surfaces (IRS) have gained much attention as an interesting framework for programmable electromagnetic-wave manipulation using low-cost, passive reflecting materials [1-3]. IRSs may provide increased spectrum efficiency, minimize interference, and enable constructive propagation of signals by adaptively adjusting the phase offsets of the passive reflecting components. Thus, IRSs have already emerged as a rather promising technology to fulfill the performance requirements of the future 6G wireless networks [4, 5]. The best IRS phase-shift design, however, turns out to be a difficult optimization problem that is non-convex and practical, having discrete-phase constraints and time varying wireless channels [6-8]. The semidefinite relaxation, the alternating optimization and the greedy selection approaches [9-11] are standard optimization techniques that might not result in a satisfactory local solution or process at a high cost.

These restrictions have been attempted to be minimised by several methods including, but not limited to, meta heuristic approaches like genetic algorithms (GA) and particle swarm optimisation (PSO) [12-14]. However, they generally require many iterations, have slow convergence and are poor scaling for increasing IRS numbers. Since they can learn environment- aware control policies without using explicit channel models, machine learning and deep reinforcement learning (DRL) have become effective techniques for IRS optimization in more recent times [15-18]. Some interesting results have been obtained in adaptive IRS control using a DRL-based approach, such as DRL-deep Q-network (DQN) [19-21], proximal policy optimization (PPO), and deep deterministic policy gradient (DDPG). Most studies so far, however, have been based on continuous-phase IRS models, on perfect channel state information (CSI), or require slow training convergence in high dimensional action spaces.

Furthermore, most recent designs rely on simpler IRS architectures, resulting in significant performance gap for discrete-phase, scalable and hardware-limited IRS designs [22-25]. These drawbacks demand an optimization method that is both fast and flexible, but model independent (not requiring any particular model of the IRS array), which is able to deal with discrete phase constraints, and which is able to function under realistic channel conditions. The last few years have seen reinforcement learning for IRS/RIS optimization to be very successful [26-28]. But there are some features, such as scalability, and then the discrete phase control, that appear to be a bit tough. In this research, an optimization framework for the DQN algorithm is introduced to try to tackle those challenges gradually.

The challenges mentioned above inspire the concept of this study, which introduces a novel discrete-phase IRS configuration optimization paradigm based on DRL. The proposed method will adaptively learn the optimum IRS phase-shift strategy in an interactive manner with the wireless environment while avoiding the need for any particular CSI or closed-form channel model. The principal contributions of this work are:

1. A model-free DQN style framework for IRS optimization is proposed, it can learn the best discrete phase-shift policy direct from doing interactions with the wireless without need for any explicit channel model or iterative optimization routines.
2. A scalable discrete-phase IRS model is developed, explicitly accounting for practical hardware constraints, making the proposed approach suitable for real-world IRS implementations.
3. A comprehensive performance evaluation is conducted, benchmarking the proposed method against Random IRS configuration, Fixed-phase IRS, Greedy selection, PSO, GA, and Traditional Beamforming techniques.
4. Significant performance improvements are demonstrated, with the proposed DQN- based approach outperforming existing optimization techniques in terms of spectral efficiency, enhanced SNR scaling with IRS size, and faster convergence.
5. Analytical and mathematical interpretations are provided to support and explain the observed empirical performance improvements.

The rest of this paper is organized as follows. The related works are given in Section 2. In Section 3, the system model is discussed. The proposed framework for DRL is presented in Section 4. A discussion of the results of the simulations is given in Section 5. Finally, we conclude this work in Section 6 by future works.

This section presents the proposed method, comprising of a system model and the framework for optimization of phase-shift by employing a DQN. It additionally offers the parameters utilized for simulation within this study.

3.1 System model

The IRS-assisted downlink wireless communication system examined in this work is described in this section. Three primary parts make up the system: A single-antenna user equipment (UE), an IRS made up of reflecting passive N pieces, and antenna base station (BS). By cleverly regulating the phase shifts of its reflecting components, the IRS is positioned between the BS and the UE to increase the received signal intensity at the user. A baseband signal x with unit average power is sent by the BS, i.e., E[|x|²] = 1. Three propagation links the BS-IRS link, the IRS-UE link, and the direct BS-UE connection affect the signal received at the UE. These channels are indicated by the symbols h_BR ∈ CN × 1, h_RU ∈ CN × 1, and h_D ∈ C, respectively. Figure 1 illustrates the architecture of the suggested IRS-assisted wireless system and shows the general structure of the BS-IRS-UE communication circuit.

1.png

Figure 1. IRS-assisted wireless communication network system model

While the IRS receives the incident signal via the channel h_BR, applies a programmed phase-shift matrix, and reflects the signal toward the UE via the channel h_RU, the BS broadcasts the signal toward the UE. Depending on the propagation environment, there may also be a direct BS-UE route, denoted by h_D. Without the use of active radio- frequency chains, the IRS functions as a largely passive met surface that modifies the wireless propagation environment. Constructive signal combining at the receiver is made possible by the independent phase adjustment of the incident electromagnetic wave by each IRS element. The effective received signal at the UE may be greatly increased by correctly constructing the IRS phase-shift vector, which will improve coverage performance, spectral efficiency, and SNR. Determining the ideal IRS phase-shift design that improves system efficiency under dynamic and time-varying channel circumstances is therefore the goal of the suggested optimization approach. It should be assert that the IRS is like a passive reflecting surface, and it does not really do channel estimation or baseband processing.

3.1.1 IRS architecture and unit cell description

The IRS is represented as a planar reflecting array made up of N sub-wavelength components, each of which can cause the incident electromagnetic wave to undergo a controlled phase shift. Through passive beamforming, the IRS allows for intelligent control of the wireless propagation environment by suitably adjusting the phase response of these components. A tiny metallic patch that reflects the incident electromagnetic wave usually makes up each IRS element, also called a unit cell. In order to provide discrete phase switching, a PIN diode is often placed between the patch and the control line. This allows the reflection phase to be changed between specified states based on the applied biasing condition. A substrate layer and a ground plane are used to provide appropriate electromagnetic isolation and reliable operation, while the control line provides the necessary DC bias from the IRS controller. A simplified IRS unit cell is schematically depicted in Figure 2, which captures the essential structural components required for system-level optimization.

2.png

Figure 2. Simplified schematic illustration of an IRS unit cell

The reflective amplitude is simply assumed to be unity for all the IRS elements, just to make the system-level analysis a little simpler, and principally to allow focusing on the degree of accuracy; the suggested DQN-based phase shift optimization procedure is successful. In real life though, practical IRS unit cells made of PIN diodes exhibit insertion losses, amplitude fluctuations, and even reflection coefficients that depend on the phase. But in this study, those hardware quirks are mostly ignored. The DQN state is the state observed at the BS at the current time interval, which is used to determine the propagation condition. The action is one of the discrete configurations of the IRS phase-shift defined in the pre-defined phase codebook. After selecting an action, it is correlated with the corresponding phase-shift vector and then applied to the IRS elements. The definitions of state and action are summarized in Table 2.

Table 2. The IRS configuration used in the simulated environment

3.1.2 Channel and IRS phase-shift model

Consider a channel model fading by quasi-static block, where the BS-IRS, IRS-UE, and direct BS-UE channels remain constant within one coherence interval and vary independently across intervals. The received baseband signal at the UE is expressed as [1, 2]

$y=\left(h_{R U}^T \Theta h_{B R}+h_D\right) x+k$        (1)

The transmitted symbol x satisfies E[∣x∣²] = 1, and k ∼ CN (0, σ²) is additive white Gaussian noise with variance σ². The IRS phase-shift matrix is defined as

$\Theta=\operatorname{diag} e^{j \theta_1}, e^{j \theta_2}, \ldots, e^{j \theta_N}$        (2)

where, θ_N is the phase shift applied by the N_th IRS reflecting element. The cascaded IRS-assisted channel is given by $h_{R U}^T$ΘhBR, and the effective BS-UE channel is expressed as

$h$eff $=h_{R U}^T \Theta \mathrm{~h}_{\mathrm{BR}}+h_D$        (3)

Given a transmit power P, the instantaneous received SNR at the UE is

$\mathrm{SNR}=\frac{P \mid h \text {eff }\left.\right|^2}{\sigma^2}$        (4)

And the achievable rate (spectral efficiency) is computed as

$R=\log _2(1+\mathrm{SNR})$        (5)

This serves as the proposed DQN agent’s training reward function and the primary optimization objective. In practical IRS implementations, each reflecting element applies a discrete phase shift selected from a finite set

$\Theta_m=\{0, \Delta \theta, 2 \Delta \theta, \ldots,(Q-1) \Delta \theta\}, m=1, \ldots, N$       (6)

where, Q = 2b denotes the number of quantization levels for a b-bit phase controller and $\Delta \theta=\frac{2 \pi}{Q}$. Consequently, the search space grows exponentially as Q^N, rendering the optimization problem highly non-convex and NP-hard even for moderate IRS sizes. The conventional optimization techniques, including alternating optimization, semidefinite relaxation and meta-heuristic algorithms, are subject to excessive complexity and limited adaptability due to the discrete phase constraint, high dimensionality and sensitivity to imperfect channel state information.

In practical IRS hardware designs can employ extra components, tunable and adjustable reflection using capacitors or varactors. In this work, the simplified unit cell model used is found to be a valid representation of the actual physical situation, as exposed through the use of amplitudes. essential reflection behavior needed for evaluation of performance and algorithm design. Table 3 listed the IRS configuration used in the simulated scenario, which includes some important characteristics of the IRS, such as the phase quantization resolution, Assumptions, spatial arrangement and the number of reflecting elements in the channel models. These settings were selected as they were found to be a good choice for evaluating the scalability and robustness of the proposed DQN. Optimize it under the specific hardware constraints and to simulate genuine deployment situations.

Table 3. The IRS configuration used in the simulated environment

It is important to note that the discrete IRS phase shift optimization issue may have a very wide action space and the problem becomes much worse with the increasing number of reflecting parts. For an IRS with (N) elements and (L) discrete phase levels, the total number of potential phase configurations explodes like (L^N). Thus an exhaustive search becomes impracticable relatively rapidly once things grow large-scale for deployments. This is somewhat addressed in the proposed DQN technique by the use of a limited phase shift codebook where each action corresponds to a possible IRS beamforming configuration, instead of a complete brute force combination phases of each single element. Moreover, the method uses experience replay along with target network updates, which helps with learning stability and also convergence. Even though the discretization we adopted is a practical compromise between complexity and spectral efficiency gains, there’s still room to go further so future work will look into more scalable DRL designs such as Double DQN, Dueling DQN, or even actor-critic style methods. These challenges motivate the adoption of a learning-based optimization framework, which is presented in the following section.

This current study looks at a kind of single-user, single-antenna IRS-assisted link scenario, basically to see how well the proposed DQN based optimization framework works, in practice. If required to push the idea further, moving toward multi-user, multi-antenna setups, and also larger scale IRS deployments seems like a key next step for future research. Also, the framework should really be read as a model-free optimization method rather than something that is strictly CSI-free, because channel estimates are still used as state observations. At the same time, there is no need for an explicit channel model, and no traditional optimization routine is required.

3.2 Proposed DRL framework

In this section, DQN proposed based framework designed to optimize the IRS phase-shift vector in real time. The overall goal is to learn a mapping between the wireless environment states and the optimal IRS configuration that maximizes the received signal strength or spectral efficiency at the UE. This section introduces the proposed DQN-based IRS optimization framework, including system modeling, state/action/reward formulation, neural architecture, and training pipeline.

3.2.1 System overview

The overall architecture of the proposed DRL-based IRS optimization framework is illustrated in Figure 3. In this system, a DQN agent is employed to autonomously control the discrete phase-shift configuration of the IRS in order to enhance the wireless link performance. The agent interacts iteratively with the wireless environment by observing the system state, selecting appropriate IRS phase-shift actions, and receiving performance- based rewards that reflect the achieved link quality at the UE. In a closed-loop interaction model, the DQN agent constantly improves its policy for making decisions depending on input from the environment. The agent applies this configuration on the reflecting surface after observing the current propagation circumstances and determining a discrete phase-shift vector for the IRS elements at each time step. A reward is created to direct the learning process once the incoming signal behavior is assessed. The agent is incentivized by this payment to choose IRS settings that optimize important performance metrics like spectral efficiency and SNR.

An extensive depiction of the environment-agent interaction is shown in Figure 3. Specifically, the agent is rewarded (rt) based on the link performance achieved after observing at time step (t) and selecting an action (at) the system state (st) that corresponds to a discrete IRS phase-shift vector. The interaction loop is completed when the environment feeds the appropriate reward back to the agent after assessing the applied action by analyzing the quality of the received signal. The suggested DRL methodology allows the wireless system to develop an effective IRS control strategy without depending on explicit CSI estimate through repeated interactions over several episodes. This learning-driven method achieves efficient and adaptable IRS phase optimization while drastically lowering computing complexity.

3.png

Figure 3. IRS phase-shift optimization using the suggested DRL framework based on DQN

The proposed IRS controller, as shown in Figure 3, is a closed-loop structure that observes, decides on, and actuates a closed-loop IRS. As seen from Figure 3, the proposed DQN based IRS controller works in a closed-loop observation-decision-actuation-feedback structure for a closed loop IRS. First, the controller reads one of the current state of the system, including the channel information received from the IRS. From this observation, the DQN agent chooses an appropriate action of phase shift distortion in the discrete action space. The selected action is then applied to the IRS elements to alter the reflected signal towards the UE. Transmission is followed by an estimation of the resulting system performance based on achievable rate/SNR, thus providing a reward back for the DQN agent. The feedback is then fed into a processing routine which adjusts future phase-shift actions and thus completes the closed-loop control cycle. The selected phase-shift architecture assumed in the considered architecture is that the DQN controller is placed at the BS-side, where system information is available, and the selected phase-shift architecture is sent to the IRS controller.

3.2.2 State, action, and reward design

State representation, action generation, reward calculation, and network training and updating make up the four primary parts of the technique. The agent and the wireless environment engage in several interaction episodes to carry out the entire learning process. The important observations for the IRS optimization job are captured by the system state represented by st, which is defined as follows: $s_t=\left\{h_{B R}, h_{R U}, h_D, \Theta_{t-1}\right\}$. The IRS phase-shift configuration used in the preceding time step is represented by $\Theta_{t-1}$. The immediate propagation environment that the wireless connection experiences is characterized by these characteristics taken together. The process at is equivalent to choosing a fresh discrete phase-shift vector. The agent outputs the phase configuration at ∈ Θ^N, where N is the number of reflecting elements, with a Q-bit phase resolution.

Even though the theoretical IRS configuration space scales in an exponential way with QN, the DQN agent that proposed doesn’t really do full, exhaustive scanning of every possible phase shift combination. Rather than brute force trying everything, it learns a phase-selection policy via ongoing back-and-forth interactions with the wireless environment, plus reward-guided updates that steer learning. That way, it can pinpoint favorable IRS configurations in a much more efficient manner, without needing to explicitly list or enumerate the entire action space. The incentive is a reflection of how the chosen IRS configuration has improved communication quality. The agent interacts with the environment at each time step t by choosing an IRS phase-shift action that influences spectral efficiency and SNR. The reward r_t is illustrated as:

$\mathrm{r}_{\mathrm{t}}=\log _2\left(1+\mathrm{SNR}_{\mathrm{t}}\right)$        (7)

The DQN agent monitors the current system state s_t, creates an action at, and applies the appropriate phase-shift configuration to the IRS at each interaction step. The learning loop is completed and progressive policy improvement is made possible by measuring the resultant received signal at the UE and feeding the computed reward rt back to the agent. The reward function is basically defined using spectral efficiency, to boost the communication performance of the IRS assisted link. The IRS controller and the biasing circuitry is assumed relatively small and nearly steady compared with the transmit power, so it is not written out in the reward formulation, explicitly though.

The channel facts needed for operation are gathered at the base station using pilot assisted estimation, and then they are relayed to the DQN agent via the IRS controller. That controller figures out the phase-shift configuration, more or less choosing the coefficients to be used. So the IRS just applies those selected terms, while all the signal processing stays on the active transceiver side.

3.2.3 DQN architecture and training procedure

The complete workflow is explained at two complementary levels in order to make the logical structure of the suggested RL framework clear. When taken as a whole, these procedures demonstrate how the agent and wireless environment interact iteratively and gradually converge on an efficient IRS phase-shift configuration.

Table 4 summarizes the main hyper parameters that were utilized to train the suggested DQN-based IRS optimization framework. These parameters control Q-network stability, learning dynamics, and experience replay efficiency periodic target network updates and experience replay are used with the target network parameters being updated as ω → ω′ every predetermined number of steps.

Table 4. DQN hyper parameters employed for training the IRS control agent

Two neural networks make up the DQN architecture used in this framework: a target Q-network Q′(s, a; ω′) and an online Q-network Q(s, a; ω). While the target network offers consistent target values throughout training, the online network is in charge of action selection. The Bellman optimality equation is used to calculate the goal value for each transition (st, at, rt, and s_t+1) sampled from the replay buffer as

$y_t=r_t+\gamma \max _{\mathrm{a}^{\prime}} Q\left(s_{t+1}, \mathrm{a}^{\prime}, \omega^{\prime}\right)$        (8)

where, s_t and s_t+1 refer to the system's present and subsequent states, respectively. at is the chosen IRS phase-shift action, rt is the reward that was earned, a′ is a candidate action in the maximization, γ is the discount factor (0 < γ ≤ 1), while ω and ω′ are the parameters of the online and target networks. The Bellman optimality rule is used to update the online Q-network parameters as expressed in Eq. (9):

$\mathrm{Q}\left(s_t, a_t, \omega\right) \rightarrow \mathrm{Q}\left(s_t, a_t, \omega\right)+\alpha\left[r_t+\gamma \max _{\mathrm{a}^{\prime}} Q\left(s_{t+1}, \mathrm{a}^{\prime}, \omega^{\prime}\right)-\mathrm{Q}\left(s_t, a_t, \omega\right)\right]$        (9)

where, α indicates the learning rate (0 < α ≤ 1), which regulates how much fresh data is added to the Q-value update. An ε-greedy strategy for exploitation and exploration strike balance, in which the agent chooses either probability ε with random action or probability 1−ε with maximizes the expected Q-value action.

$a_t=\left\{\begin{array}{c}\text { random action, with probability } \varepsilon \\ \arg \max _a Q\left(s_t, a_t, \omega\right), \text { with probability } 1-\varepsilon\end{array}\right.$

4.png

Figure 4. Flowchart of the proposed DRL training -based IRS phase-shift optimization algorithm

Figure 4, which displays the algorithm-level flowchart of the suggested DRL-based IRS phase-shift optimization technique, provides an illustration of the training framework.

Additionally, Figure 4 provides a comprehensive overview of the episode-based training loop, including the network update process, replay buffer operation, and agent-environment interaction.

3.2.4 Training workflow

The learning agent and the wireless environment interact iteratively during the training process of the suggested DQN-based IRS optimization framework. The starting state, which includes the applied IRS configuration and the current channel circumstances, is then obtained by resetting the environment. Using an ε-greedy strategy that strikes a balance between exploration and exploitation, the agent chooses an action based on this state. A discrete IRS phase-shift configuration is determined by the chosen action and applied to the reflecting surface. The wireless environment then changes into a new state and provides a scalar reward that represents the link performance that was attained. The replay buffer, which functions as the agent's experience memory, contains the resulting transition tuple (st, at, rt, s_t+1). Gradient descent is used to update the online Q-network's parameters by randomly sampling mini-batches of previous transitions from the replay buffer. For stability of training, the target Q-network is periodically synced with the online network. This interaction loop is repeated for a number of time steps and training events until the learning process converges.

Pseudo-code presents the pseudocode of the suggested DQN-based IRS optimization algorithm, summarizing the whole training process and algorithmic flow. The pseudocode describes the following: initialization, agent environment interaction, experience storage, updates of the Q-network, and the convergence process.

In this formulation, the iterative learning mechanism by which the agent is able to converge to an effective IRS phase shift control policy is presented in a simple and repeatable manner.

3.2.5 Simulation setup

The simulation environment is optimized by using a DQN-based controller to determine the configuration of the IRS, which takes into account an IRS-assisted wireless communication link. The efficiency of the Numerical simulation is used to evaluate the suggested learning-based optimization method in a typical downlink IRS-assisted communication scenario. The IRS-DRL experiments' simulation parameters are shown in Table 5. The carrier frequency, bandwidth, transmit power, propagation lengths, IRS characteristics, and RL settings are all specified by these parameters. The selection of Communication related and learning related criteria is carefully selected in order to obtain a fair evaluation and simulate real deployment scenarios.

Table 5 shows an overview of the simulation parameters of the IRS-DRL architecture. Selected Settings define the wireless environment, IRS setup and channel circumstances used for the performance assessment. The setups are useful for testing the effectiveness of the proposed DRL-based phase-shift optimization technique. In addition, the simulation providing a solid basis for comparisons with the benchmark improvement methods that covered in the following section.

Table 5. Simulation parameters for the IRS-DRL experiment