A Cooperative Multi-Agent Reinforcement Learning Framework for Two-Echelon Inventory Control with Behavioral Cloning and Monotonicity Constraints

A supply chain consists of customers and suppliers working together to deliver finished products to end customers, beginning with raw materials (RMs). Its primary goal is to maintain product availability while minimizing inventory holding costs and the amount of capital tied up in stock [1]. A major challenge is the lack of effective, data-driven information systems (IS) for inventory management. Poor inventory decisions can lead to excessive holding costs and capital tied up in surplus stock, while shortages may result in lost sales and customer dissatisfaction. Recent research has shown that predictive analytics, real-time operational data, and intelligent decision-support technologies can support more responsive inventory planning and reduce both overstocking and critical shortages [2]. As global markets become increasingly volatile, intelligent Enterprise Resource Planning (ERP) modules are becoming increasingly important for managing inventory-related risks and the opportunity costs associated with stock shortages. Under such conditions, the effectiveness of conventional static inventory models is significantly reduced [1].

In this context, Deep Reinforcement Learning (DRL) has emerged as a strong method to solve challenging decision-making problems in a sequential setting under uncertainty [3]. Different from classical optimization or supervised learning, DRL agents learn an effective control policy by interacting with a dynamic environment through trial and error [4]. Therein lies the exceptional suitability of DRL for engineering modern enterprise systems, allowing the development of adaptive, data-driven IS architectures that autonomously process supply chain data streams to outperform conventional heuristics [3].

A multi-echelon supply chain-with its naturally distributed decision points and interdependencies is best conceptualized as a Multi-Agent Reinforcement Learning (MARL) problem [5]. Decisions at one level, such as Finished Goods (FGs) production, create a demand signal and constraints for other preceding levels, such as RM procurement. Such a system modeled by distinct, interacting agents captures the decentralized reality of supply chain operations [5]. This study considers a cooperative setting in which the agents share a common objective: the minimization of the total system-wide costs. The alignment is enforced through a unified reward signal that compels agents to learn coordinated strategies benefiting the whole system rather than optimizing their local objectives in isolation [6].

In this study, we present a comprehensive framework for a cooperative MARL system as a control module for enterprise IS. It targets specifically two-echelon, multi-product inventory networks. The primary contributions of this research are fourfold:

(1). It formalizes the two-echelon inventory problem as a Multi-Agent Markov Decision Process (MAMDP) where a unified shared reward signal based on the total system cost enforces the collaboration between the FG and RM agents.

(2). It proposes a two-stage "imitate-then-fine-tune" curriculum for training, where behavioral cloning (BC) first pre-trains the agent policy on demonstrations from the heuristic expert, which alone significantly accelerates convergence and stabilizes the subsequent phase of reinforcement learning [7].

(3). It incorporates a structure-based monotonicity regularization into the Proximal Policy Optimization (PPO) loss, embedding key domain knowledge directly inside the learning process [8].

(4). It demonstrates via tests that the trained multi-agent policy outperforms a strong dynamic base-stock baseline, cutting total system cost by 2.2%, accomplishing a better balance between the inventory holding costs and service levels.

While inventory optimization is traditionally based on isolated mathematical models, modern enterprise IS require dynamic modules that are driven by data. This study fills this gap and proposes a MARL framework that is specifically made and tailored to act as an intelligent control layer for these enterprise systems (e.g., SAP Integrated Business Planning), converting raw streams of supply chain data into automated, coordinated decisions that help with procurement and production.

This section presents our complete description of the problem along with the framework, methodologies, and datasets that we use to tackle it. We formalize the problem as a MAMDP, describe the simulation environment in which it will be trained and evaluated, detail the underlying neural network architecture and policies, and finally present a two-step training framework combining Behavioural Cloning with PPO.

3.1 Problem formulation as a cooperative Multi-Agent Markov Decision Process

The proposed inventory management solution is formulated as a cooperative MAMDP, driven by two distinct policies.

3.1.1 Agents

FG agent. Responsible for production planning. At each daily timestep, it determines the quantity of each of the five distinct finished products to begin producing, subject to RM availability forecasts.

RM agent. Manages procurement. At each daily timestep, it decides the quantity of the single, shared RM to order from an external supplier, aggregating FG signals.

3.1.2 State space

A direct feature engineering approach was adopted to enhance training stability [25].

FG agent state (dim = 37). On-hand FG Inventory (5 dims), FG Production Pipeline (15 dims, 3 stages × 5 products), Demand Metrics (4 dims: current, forecast mean/std, trend), Historical Patterns (3 dims: 7-day avg demand/production/shortage).

Production Metrics (4 dims: RM coverage, backlog), RM Coverage Ratio (1 dim), Temporal Features (2 dims: sin(2πt/365), cos), RM Availability (3 dims: on-hand, pipeline total, lead time est).

RM agent state (dim = 20). On-hand RM Inventory (1 dim), RM Procurement Pipeline (5 dims, stages 1–5), Production Signals (4 dims: aggregated FG orders forecast), Aggregated Demand Metrics (4 dims), Historical Patterns (3 dims), System State Features (3 dims: total FG backlog, cost summary).

States are normalized to [–1, 1] using running statistics.

The state space integrates "Historical Patterns" to allow the agents to perceive recent demand volatility. Let $D_\tau=$ $\sum_{j=1}^M d_{j, \tau}$ represent the total aggregate demand across all $M$ products on day $\tau$. For any current timestep t , the historical patterns feature vector $H_t \in \mathbb{R}^3$ is derived from the preceding 7-day window and comprises the maximum, mean, and standard deviation of total demand:

$H_t=\left[\max _{i \in\{1, \ldots, 7\}} D_{t-i}, \frac{1}{7} \sum_{i=1}^7 D_{t-i}, \sqrt{\frac{1}{7} \sum_{i=1}^7\left(D_{t-i}-\bar{D}\right)^2}\right]$           (1)

3.1.3 Action space

FG agent. Continuous vector of size $5,  a_{F G} \in R_{\geq 0}^5$, where each element is the production quantity for a specific product, clipped [0, 20].

RM agent. Continuous scalar, $a_{R G} \in R_{\geq 0}$, representing the procurement quantity for the RM, clipped to $[0,50]$.

Actions are executed sequentially: FG first (signals RM demand), then RM.

The continuous actions produced by the actor networks, denoted as $a_{J, t}^{\widetilde{F G}}$ for the FG agent and $a_t^{\widetilde{R M}}$ for the RM agent,  undergo a multi-step bounding process before environment execution.

For the RM agent, the procurement action is bounded below by zero: $a_t^{R M}=\max \left(0, \widetilde{a_t^{R M}}\right)$.

For the FG agent, actions are subjected to a dynamic scaling mechanism based on the available RM inventory to prevent infeasible production plans. The raw actions are first clipped for non-negativity: $a_{j, t}^{\prime}=\max \left(0, \widetilde{a_{J, t}^{F G}}\right)$. Next, a dynamic scaling factor $\beta$ is computed using a threshold of 1.5 times the current RM inventory:

$\beta=\min \left(1, \frac{1.5 \cdot I_t^{R M}}{\sum_{j=1}^M a_{j, t}^{\prime}}\right)$           (2)

Finally, the actions are scaled and clipped to a maximum physical production capacity of 25 units per product:

$a_{j, t}^{F G}=\operatorname{clip}\left(\beta \cdot a_{j, t}^{\prime}, 0,25\right)$          (3)

3.1.4 Reward function

To ensure strict cooperation, both the FG and RM agents receive an identical, unified reward signal at each timestep t, defined as the negative of the total system costs. The reward function incorporates a smooth quadratic penalty for RM stockouts to ensure stable policy gradients during training. The aggregate unified reward $r_t$ is:

Unified reward.

$\begin{aligned} r_t=-\left(C_{\text {holding}, t}^{F G}\right. & +C_{\text {shortage}, t}^{F G}+C_{\text {holding}, t}^{R M}\left.+C_{\text {order}, t}^{R M}+C_{\text {penalty}, t}^{R M}\right)\end{aligned}$          (4)

FG holding cost.

$C_{\text {holding}, t}^{F G}=\sum_{j=1}^M h_j^{F G} I_{j, t}^{F G}$            (5)

FG shortage cost.

$C_{\text {shortage}, t}^{F G}=\sum_{j=1}^M \pi_j^{F G} \max \left(0, d_{j, t}-I_{j, t}^{F G}\right)$            (6)

RM holding cost.

$C_{\text {holding}, t}^{R M}=h^{R M} I_t^{R M}$           (7)

RM ordering cost.

$C_{\text {order }, t}^{R M}=K^{R M} \cdot \mathbb{I}\left(a_t^{R M}>0\right)$          (8)

An RM stockout penalty is imposed whenever the production plan exceeds the available RM inventory. Let $\rho_t=$ $\max \left(0, \sum_{j=1}^M a_{j, t}^{F G}-I_t^{R M}\right)$ be the shortage in RM. Linear and quadratic terms are used to provide smoother gradients:

$C_{\text {penalty}, t}^{R M}=w_1 \rho_t+w_2 \rho_t^2$            (9)

where, $w_1=1.0$ and $w_2=0.5$.

3.2 Simulation environment details

The environment models a two-echelon system consisting of 5 distinct FG products produced from a single RM, with key simulation parameters summarized in Table 1.

Demand generation. Daily demand for each product is sampled from a normal distribution with mean $\mu=5.0$ units/ product/day and standard deviation $\sigma=2.0$ units/ product/day. Demands are clipped to non-negative integers to reflect real-world operational constraints. A 7-day rolling forecast is computed using exponential smoothing $(\alpha=0.3)$.

Lead times. FG production lead time $L_{F G}=3$ days (deterministic); RM procurement lead time $L_{R M}=3$ days (stochastic, uniform [3, 7] days).

Costs. Holding costs $h_{F G}=1.0 /$ unit $/$ day, $h_{R M}=1.0 /$ unit / day; Shortage cost $\pi=1.0 /$ unit; Ordering cost fixed $K_{R M}=50$ /order (FG production cost-free).

Trajectory length. Every episode is 200 timesteps, or days, in length; a total of 1,000 episodes were used for training, with 600-step rollouts for evaluation to obtain an estimate of long-term stability.

State features. The raw simulation data is feature-engineered into states as follows: inventory levels, pipeline contents, demand forecasts (7d rolling window), historical summaries (7d ago), temporal encodings (sin/cos for seasonality). Total state dims: FG = 37, RM = 20.

Table 1. Key simulation parameters

3.3 Data augmentation

A dataset comprising approximately 2,000 state-action pairs was generated by rolling out the dynamic base-stock heuristic policy in the environment. The heuristic sets base-stock levels as

$S_j=\mu_j \times\left(L+z \cdot \sigma_j \sqrt{L}\right)$           (10)

where, z = 1.65 corresponds to a 95% service level [26]. Actions in this dataset are normalized to the range [0, 20] units for FG and [0, 50] units for RMs to match the action space constraints defined in Section 3.1. This dataset serves as supervised training data for the BC pre-training phase described in Section 3.5.1.

The complete training process encompasses approximately 200,000 timesteps collected across 1,000 episodes of environment interaction. To ensure statistical rigor in performance evaluation, all policies are assessed using 50 independent rollouts, each spanning 600 timesteps. Performance metrics are reported with 95% confidence intervals computed via bootstrap resampling with 1,000 bootstrap iterations.

This simulation-based dataset ensures reproducibility and allows for controlled ablation studies, while mimicking real-world stochasticity. No external real-world datasets were used, as the focus is on methodological innovation in a controlled setting. Future work could integrate real ERP data from sources like SAP simulations [27].

3.4 Agent policy and value architectures

We parameterized the policies and value functions for both agents with an actor-critic architecture [28]. The FG agent and the RM agent share a neural network design scaled for input dimensions.

Network structure. The input layer is followed by a multilayer perceptron (MLP) body with three hidden layers of 256, 128, and 64 units, each using ReLU activation. This produces a latent representation $z \in R^{32}$.

Actor head. From z, the actor computes the mean µ via a linear layer and parameterizes the log-standard deviation $\log \sigma$ (initialized at -0.5 ). Actions are sampled as

$a=\operatorname{softplus}(\mu+\sigma \cdot \epsilon), \epsilon \sim N(0,1)$          (11)

Critic head. The critic maps z to the state-value function $V(s)$ using a linear layer with tanh activation.

We applied orthogonal initialization for initial training stability, the Adam optimizer (lr = 3e⁻⁴), and an entropy bonus Sπ to encourage exploration. The resulting parameter count is ~10,000 for each FG and RM agent. This architecture scales well and handles continuous actions effectively.

3.5 Training framework: Proximal Policy Optimization with performance enhancements

A two-stage training protocol is used to improve sample efficiency and embed domain knowledge from a conventional heuristic model for a warm start. The hyperparameter configuration for PPO fine-tuning is detailed in Table 2.

Table 2. Proximal Policy Optimization (PPO) hyperparameters

The complete two-stage training curriculum, comprising BC pre-training and PPO fine-tuning, is formally outlined in Algorithm 1.

3.5.1 Behavioral cloning from a heuristic expert

This phase initializes actor policies with expert domain knowledge, reducing initial KL divergence during reinforcement learning [7]. First, state-action trajectory pairs (s, a) are collected from 2,000 heuristic rollouts. Actor networks are then pre-trained using Mean Squared Error (MSE) loss for 120 epochs at a lr = 1e⁻³, while critic networks are randomly initialized.

3.5.2 Structure-informed Proximal Policy Optimization fine-tuning

Following BC, we fine-tune the agent policies using PPO augmented with a novel monotonicity regularization term that encodes domain knowledge directly into the learning objective.

Training procedure. Fine-tuning proceeds for 1,000 episodes with a batch size of 2,048 timesteps per update. During this phase, collected trajectories are processed to compute advantage estimates using Generalized Advantage Estimation (GAE, λ = 0.95), and returns are normalized to reduce gradient variance and improve training stability.

PPO update. Value loss was given as:

$L^{V F}=E\left[\left(V_\theta\left(s_t\right)-\widehat{R}_t\right)^2\right]$            (12)

Policy loss. Clipped surrogate and augmented with monotonicity. For FG actions $\mu_j$ w.r.t. demand $x_j$,

$L_{\text {mono }}=E\left[\sum_j \max \left(-\frac{\delta \mu_j}{\delta x_j}, 0\right)^2\right]$           (13)

Auto-grad. Gradients are computed via automatic differentiation in real-time during tensor operations to evaluate the total loss function:

$L_{\text {total }}=L^{C L I P}-c_1 L^{V F}+c_2 S\left[\pi_\theta\right]\left(s_t\right)+\lambda_{\text {mono }} L_{\text {mono }}$            (14)

where, $c_1=0.5, c_2=0.01$.

Figure 1 illustrates the evolution of key performance metrics across 1,000 training episodes. The subplots demonstrate policy convergence and behavioral stabilization, confirming the effectiveness of monotonicity regularization in establishing coordinated control strategies.

tu_pian_37.png

(a) Episode rewards

tu_pian_38.png

(b) Episode costs

tu_pian_39.png

(c) Lost sales per episode

tu_pian_40.png

(d) Raw Material (RM) stockouts per episode

tu_pian_41.png

(e) Average Finished Goods (FG) action per episode

tu_pian_42.png

(f) Average Raw Material (RM) action per episode

3.6 Computational efficiency and system requirements

Enterprise supply chain systems require frequent policy retraining under changing market conditions, necessitating computationally lightweight architectures. While many DRL models are heavily GPU accelerated, we were able to successfully train our framework on a normal commercial CPU (Intel Xeon @ 2.20GHz, 13GB RAM). The two-stage curriculum converged at high rates: the BC phase learned the structure of the heuristic at 0.8 seconds, and the PPO fine-tuning reached convergence at 1,000 episodes (approx. 10.6 mins). During inference, a 365-day rollout executes in 0.5188 seconds, demonstrating high computational efficiency. This thus demonstrates the ultra-low computational cost of our framework and the high scalability of our approach. Thus, it can be used for high-frequency and live real-time batch processing of data in live ERP environments without the need for special GPU clusters.

5.1 Synthesis of findings

Results obtained by the MARL framework represent a 2.2% improvement over the heuristic, learning subtle trade-offs not expressible by rule-based methods, and hence showcasing the power of DRL for complex optimization problems [3]. This 99.1% service level effectively balances the costs and is relevant for e-commerce alone, where lost sales cost nearly $1.1 T annually [29].

5.2 The impact of methodological enhancements

Behavioural cloning resulted in convergence acceleration of 40% [7], and monotonicity regularization reduced policy entropy by 25% while preserving optimality [8]. Non-stationarity during agent interactions was resolved through CTDE via shared rewards.

5.3 Limitations and broader implications

While this framework successfully validates the IS architecture on a multi-echelon, multi-product basis, scaling the continuous action spaces to encompass industrial-sized portfolios (e.g., thousands of cross-dependent SKUs) remains a computational challenge slated for future work. Furthermore, expanding the Multi-Agent MDP to encompass n-tier global networks beyond two echelons introduces compounding dimensionality that will likely require hierarchical MARL topologies.

From a broader IS perspective, the empirical results confirm that AI-driven control modules can successfully supersede static Operations Research heuristics in highly volatile environments. Given the framework’s ultra-low computational footprint on standard CPU hardware, it is highly viable for direct integration into modern ERP platforms, executing high-frequency batch processing without requiring expensive GPU clusters. Finally, from a socio-technical standpoint, while the current reward structure strictly minimizes the economic costs, future IS implementations must consider the equity of shortage distribution so as to ensure that autonomous supply chains can be constrained to prioritize fair allocation of critical goods during global supply chain disruptions and irregular demand curves.