Multimodal Sensor Fusion for Waste Management Using Graph Neural Networks

With the rapid growth of urban populations and industrial activities come increased levels of solid waste, creating major worldwide environmental, health, and economic challenges. Proper waste management is needed to reduce environmental pollution, improve resource efficiency, and support sustainability. Waste classification and collection rely heavily on traditional manual and labor-intensive approaches, which are often insufficient in size and complexity in today's urban landscape. The recent emergence of advanced sensor technologies and machine learning (ML) techniques could offer exciting solutions to help transform waste management systems through automation and intelligent decision-making. Deep learning (DL) methods have been shown to be effective in waste classification with a good degree of accuracy for the identification of different waste categories from image and sensor data. Rayhan and Rifai [1] utilized convolutional neural networks (CNNs) for a multiclass waste classification system and produced better classification accuracy than traditional methods. Similarly, Nahiduzzaman et al. [2] produced an automated waste classification system based on deep learning techniques to assist with efficient recycling and sustainable waste management. Chhabra et al. [3] proposed an intelligent waste classification approach based on an improved multilayered CNN, overcoming issues associated with feature extraction and model generalization.

Further research has investigated object detection frameworks for real-time waste classification tasks, such as You Only Look Once (YOLO). Riyadi et al. [4] conducted a comparative study of YOLO models to assess potential for recyclable waste detection and found acceptable inference speed. Abu-Qdais et al. [5] created a versatile solid waste classification system based on image processing techniques and ML algorithms, providing some practical insight for implementation-related discussions of urban management. The inclusion of multi-scale context information can improve detection performance in heterogeneous settings. Li and Zhang [6] created a multi-scale context fusion network based on urban solid waste detection from remote sensing images, demonstrating the importance of spatial features at multiple resolutions. Interest in multimodal sensor fusion is also growing, allowing systems to leverage acoustic or environmental or other sensory information as context to improve classification. Duan et al. [7] provided examples of multimodal sensors used with ML fusion approach in robotics that provide the foundation for development of adaptive and resilient systems for waste monitoring.

Recent studies have continued to apply multimodal fusion approaches for classification and bulky waste presents many of the same difficulties with noisy incomplete data. Bihler et al. [8] introduced a deep learning-based multi-sensor data fusion system for classification of bulky waste images, and demonstrated enhanced robustness and scalability. Likewise, plastic waste detection has been discussed by Kunwar et al. [9] by providing a dataset and work on plastic waste classification using deep learning. A comparative review of deep learning models for waste classification was conducted by Qiao [10], which highlighted the importance of proper architecture and preprocessing models. The efforts to develop methods for sustainable waste recycling include Narayan’s [11] DeepWaste framework, which utilizes deep learning for waste classification with sustainability focus. Jin et al. [12] presented a machine vision system for garbage detection, integrating deep learning and new feature extraction algorithms to overcome a challenge faced by industry in waste sorting and recycling.

While multimodal sensor fusion has been shown to improve the accuracy of waste monitoring solutions, most existing approaches treat the waste bin and sensing unit as separate and independent entities. This fails to capture the inherent relational structure of how waste is disposed of and managed within urban systems. Typically, the waste bin is connected to other waste bins through physical proximity, road networks, similar collection routes, and similar usage patterns driven by human activity and the layout of the city. Graph neural networks (GNNs) provide a well-defined means of explicitly modeling the networks of interdependencies between waste disposal and collection devices, defining each waste disposal or collection unit as a node and the respective spatial-functional relationships between them as edges.

Unlike conventional Deep Learning models that rely on grid-based or sequential data inputs, GNNs allow localized message passing and enable each waste bin to tailor its predictions based on the fill states of neighboring waste bins, previous patterns of use, and projected collection times. This capability is especially important for waste management situations characterized by non-uniform generation of waste, rapidly changing patterns of use, and effluent collection cost based on route, where isolated predictions can lead to redundant and suboptimal collection decisions. The new methodology combines both graphical methods for representation and analysis of multimodal signal input to gain from their strengths to improve load level forecasting accuracy, anomaly detection reliability collection route optimization. Additionally, the graph-centric approach serves both as the primary rationale behind this work and as a reason for differentness from existing multimodal systems of waste management which do not incorporate relationship modelling directly.

Furthermore, GNNs provide automatic support for spatio-temporal reasoning, thereby making it possible to jointly model accumulating temporal waste and interacting spatially bins. This capability is essential to achieving optimized waste collection on a city-wide scale in real time. The rest of the paper is organized as follows: Section II discusses previous research conducted by many researchers. In Section III, the proposed GNN-MSF framework describing the system model, formulation of the mathematical model, and objectives for multi-objective research. Section IV includes explanations for the set-up for computational experiments, simulation scenarios, and performance metrics, followed by a comprehensive results and discussion section comparing the proposed model to baseline designs on key measures. Section IV identifies important findings and contributions with implications for future research.

The proposed model, GMSF-GNN, aims to combine heterogeneous sensor modalities (visual, acoustic, environmental) for reliable waste classification and anomaly detection. It models spatial and temporal relationships across sensor locations via a Graph Neural Network, and operate under realistic constraints (asynchronous/missing data, limited edge compute). The system consists of four main modules also shown in Figure 1.

• Modality-Specific Feature Extractors (MSFE)
• Adaptive Multimodal Fusion with Attention (AMFA)
• Graph Construction & Graph Neural Network Module (GNNM)
• Task Heads: Classification & Anomaly Detection (TH)

Figure 1. Overall architecture diagram

3.1 Modality-specific feature extractors

The framework put forward here uses MSFE to derive rich and discriminative representations from each of the data modalities prior to formidable graph-based fusion. Each modality has a lightweight feature extractor designed for edge deployment.

Now let us think about the three modality-specific inputs at node i and time t: K-dim IoT sensor vector, image, and raw audio waveform (or pre-windowed signal). Eq. (1) describes the raw input data in multimodal IoT contexts taking the form of image, audio, and IoT sensor readings.

$\begin{gathered}X_i^{(i m g)}(t) \cdot \in R^{H \times W \times C}, X_i^{(a u d)}(t) \in R^{T_{a u d}} \cdot X_i^{(i o t)} \\ (t)=\left[x_{i, 1}(t), \ldots, x_{i, K}(t)\right] \top \in R^K\end{gathered}$                   (1)

Here, $X_i^{i m g}$ is the image with height H , width W , and channels $\mathrm{C}, X_i^{ {aud }}(t)$ is an audio signal with length Taud, and $X_i^{{iot }}(t)$ is the IoT sensor vector which is K-dim IoT sensor data. Also, we utilize Eq. (2) to extract spatial features from the waste images input or underlying raw image data while adapting the image features to a resource-constrained IoT application.

$\hat{F}_i^{(i m g)}(t)={ResAdapter}\left({Attn}_{s p}\left(\phi_{C N N}\left(X_i^{(i m g)}(t)\right)\right)\right)$                   (2)

Here, the image is first passed through a CNN feature extractor, $\phi_{CNN}$, which extracts or encodes the image information. Finally, Eq. (3) computes attention weights that focus uniquely on the salient regions of the image and reduce the weight of less informative regions, effectively concentrating on the most relevant parts of the image context.

$\begin{gathered}A_{s p} \sigma\left({Conv}_{1 \times 1}\left({Pool}\left(\phi_{C N N}(1)\right)\right)\right), {Attn}_{s p} \\ (Z)=A_{s p} \odot Z\end{gathered}$               (3)

Here, the spatial attention mask is denoted as A_sp, it is formed by pooling CNN features and applying a 1 × 1 convolution followed by sigmoid activation σ. It is then applied element-wise ⊙ to the features Z reinforcing attention to the discriminative occurrences in waste image. This block is relatively lightweight since it requires tuning down features for a specific dataset, obviating the need to retrain the full CNN as given in Eq. (4). 

${ResAdapter}(Z)=Z+\gamma \cdot {ReLU}\left(W_{a d} Z+b_{a d}\right)$                  (4)

Here, W_ad, b_ad are learnable parameters, γ is a scaling term, and the ReLU works to promote non-linearity, facilitating lightweight adaptability. The conversion of raw audio into time-frequency representation and the extraction of higher-level embeddings are shown in Eq. (5). 

$\hat{F}_i^{(a u d)}(t)=\phi_{A U D}\left({Mel}\left(X_i^{a u d}(t)\right)\right)$               (5)

Here, $\phi_{AUD}$ represents audio signals that are first converted to Mel-spectrograms, which represent the human ability to discriminate auditory information better. $X_i^{a u d}(t)$ denotes the audio feratures. Eq. (6) is used to extract temporal trends in the IoT readings.

$\widehat{F}_i^{(i o t)}(t)=\phi_{I O T}\left(x_i, \cdot(t-T+1: t)\right)$                (6)

Here, $\phi_{I O T}$ denotes the IOT readings from the prior T timesteps that are processed by a temporal encoder. This incorporates temporal dynamics, thus capturing the complete temporal dynamics of the bins creating robust temporal embeddings $F^{i(i o t)}(t)$. Eq. (7) continues to examine the modality features, projecting them into one common space while still outputting uncertainty (σ).

$\tilde{F}_i^{(m)}(t)=P_m{{\hat{F}_i}}_i^{(m)}(t)+q_m, \sigma_i^m(t t)={SoftPlus}\left(U_m \widehat{F}_i^{(m)}(t)+c_m\right)$                 (7)

Here, $F_i^{(m)}(t)$ denotes all modality embeddings are projected into the mean vector and $\sigma_i^m(t)$ denotes the uncertainty vector. The SoftPlus function used to guarantee the uncertainty's positivity. This essentially means that you have probabilistic embeddings that take modality reliability into account. Each of these feature vectors is then modeled probabilistically as a Gaussian distribution N(μ, Σ), based on Eq. (8).

$\begin{gathered}\left(\mu_i^{(m)}(t), \Sigma_i^{(m)}(t)\right), \mu_i^{(m)}(t)=\tilde{F}_i^{(m)}(t), \Sigma_i^{(m)}(t)= {diag}\left(\sigma_i^{(m)}(t)\right)\end{gathered}$                   (8)

Here, each modality embedding is modeled as the Gaussian distribution with mean denoted as $\mu_i^m(t)$ and covariance denoted as $\Sigma_i^m(t)$. Eq. (9) handles the potential of a missing modality (e.g. possibility the IoT has failed, or it doesn't capture audio, etc.).

$\delta_i^m(t)=\left\{\begin{array}{lr}1 & { if }\ m \epsilon(i, t) \\ 0 & \text { otherwise }\end{array}\right.$                (9)

Here, the binary indicator captures the fact that it is tracking whether or not the modality is available or missing. If the IoT sensor happens to not report values. Eq. (10) learns a single confidence score (0-1) for each modality based on its features, uncertainty and availability.

$\begin{gathered}r_i^m(t)=\sigma\left(w_r^{\top} g_i^m(t)+b_r\right), g_i^m(t)=\left[\mu_i^m(t) \|\right. \left.\log \left({diag}\left(\Sigma_i^{(m)}(t)\right)\right) \| \delta_i^m(t)\right]\end{gathered}$                 (10)

Here, a reliability gate computes rim(t) using the modality statistics. $R_i^m(t)$ takes into account the mean embedding, log-variance, and whether the modality is available. Finally, Eq. (11) describes randomly dropping modalities during training to help improve robustness against failing sensors.

$\tilde{\delta}_i^m(t)={Bernoulli}\left(p_{ {keep }}^m\right), \delta_i^m(t) \leftarrow \delta_i^{(m)}(t) \cdot \tilde{\delta}_i^m(t)$              (11)

Here, A modality is dropped with probability $p_{{keep }}^m$. The final normalized embedding per modality is formulated in Eq. (12), which also weights by reliability and availability.

$\bar{F}_i^m(t)=r_i^{(m)}(t) \cdot \delta_i^{(m)}(t) \cdot \frac{\mu_i^m(t)}{\left(\left\|\mu_i^m(t)\right\|_2+\epsilon\right)}$                  (12)

Here, the modality embedding is gated through the reliability score $r_i^m(t)$ and an availability indicator $\delta_i^m(t)$, and is then L2-normalized for scale bias. In Eq. (13), we have inter-modality consistency loss that enforces agreement across modalities.

$\begin{gathered}L_{ {cons }}=\frac{1}{|P|} \sum_{(m, n) \in P} E_{i, t}\left[\delta_i^{(m)}(t) \delta_i^n(t) \cdot \| \bar{F}_i^m\right. \left.(t)-\bar{F}_i^m(t) \|_2^2\right]\end{gathered}$                (13)

To improve alignment across modalities, we again use hooks to enforce a consistency loss when there are embeddings from different modalities available. The student network model based on the concept of a knowledge distillation technique in which smaller and less powerful models (students) learn to approximate the output of more powerful models (teachers). In this context, the individual modular extractors are the teacher networks and the student projection takes a high-dimensional vector of embeddings and projects them down into a lower-dimensional space for use in edge processing. The methodology here also differs from how the traditional image classification techniques implemented knowledge distillation at the output level, in that the proposed methods allows for feature level knowledge transfer while also maintaining cross-modal correspondence, yet minimising overall processing and memory load. The proposed frameworks will allow for efficient multi-modal fusing and construction of the graph representations in a resource-constrained environment without having to retrain or modify the original teacher networks. Eq. (14) modifies the features for low power IoT devices via a student network (ds < d).

$\widehat{F}_{i, { student }}^m(t)=S_m\left(\widehat{F}_i^m(t)\right)$                (14)

By embedding across low dimensionality, we see with each modality that we derive a normalized embedding for the fusion layer. Each modality contributes an embedding, uncertainty estimate, and presence indicator as shown in Eq. (15).

$\left.F_i^{(m), { out }}(t)=\bar{F}_i^m(t), \Sigma_i^m(t), \delta_i^m(t)\right)$                 (15)

Here, the final output for each modality provides the layer ith normalized embedding is denoted as $\bar{F}_i^m(t)$, uncertainty estimate referred to as $\Sigma_i^{(m)}(t)$, and the availability indicator as $\delta_i^m(t)$.

3.2 Adaptive Multimodal Fusion with Attentionwith attention  

The fusion process is adaptive: If the image modalities are reliable, αimg is large; If the IoT modalities are noisy or missing altogether, α_iot is quite low. The output of the fusion layer has access to both the content and the confidence of each modality. At the core of the fusion layer is a learnable adaptive attention scoring function that determines the weight of each modality using Eq. (16).

$\begin{aligned} e_i^m(t)= & v \top \tanh \left(W_f \bar{F}_i^m(t)+U_f \delta_i^m(t)-\right. \left.V_f \log \left(\Sigma_i^{(m)}(t)+\epsilon\right)\right)\end{aligned}$                  (16)

Here, $W_f, U_f, V_f$ are transformation matrices, $\delta_i^m(t)$ ensures a non-weight is applied to missing modalities, $-\log \left(\sum_i^m(t)\right)$ tells us that higher levels of uncertainty leads to a lower weight with attention scoring function, and v is the projection vector to obtain the scalar score. Overall, this guarantees that reliable modalities drive the output of fusion, while noisy or missing modalities are down weighted as shown in Eq. (17).

$\alpha_i^m(t)=\sum_{n \in M} \exp \left(e_i^n(t)\right) \exp \left(e_i^m(t)\right)$                     (17)

Here, the attention weights, denoted as $\alpha_i^m(t)$, act as a probability distribution over the available modalities. This enforces an adaptive balance, such that image is deemed more as shown in Eq. (18).

$\hat{F}_i^{\text {fusion }}(t)=m \in M \sum \alpha_i^m(t) \cdot \bar{F}_i^m(t)$                 (18)

The final fused embedding becomes a weighted sum of the embedding’s modalities, where attention dynamically selects the values of the weights. The model has the ability to adaptively shift focus onto one modality, depending on the waste type e.g., Glass bottle → audio (breaking sound) + image, Food waste → IoT (weight, fill level), Plastic waste → visual features dominate as given in Eq. (19). 

$F_i^{ {final }}(t)=F_i^{ {fusion }}(t)+\lambda \cdot m$               (19)

Here, $r_i^m(t)$ denote the reliability gate from MSFE and λ will denote influence factor over the fusion. This ensures that even if attention dynamically allows for the majority portion of attention to misallocate weights, there will still be a direct influence from weight on the fused feature.

3.3 Graph construction

The system is defined as a dynamic graph $G t=\left(V, E_t, W_t\right)$, where the nodes (V) are defined such that each node $v_i \in V$ represents an entity. Edges $\left(E_t\right)$ denote spatial or temporal relationships between nodes. Weights $\left(W_t\right)$ is a combining static dynamic edge strength. There are three steps to constructing a weighted graph: spatial proximity, dynamic similarity, and temporal continuity. The graph structure captures spatial proximity and temporal relationships. We first compute a static adjacency matrix that incorporates physical closeness as given in Eq. (20).

$w_{i j}^0=\exp \left(-\frac{{dist}(i, j)}{\sigma}\right)$            (20)

Here, dist(i,j) is a Euclidean distance between nodes i and j based on coordinates, σ is a scaling factor that adjusts how sensitive the measure is to closeness, and $w_{i j}^0$ is the base weight which is stronger the closer two nodes are. Not only do node states change over time but we also have dynamic edge weights that account for how similar nodes are to each other as shown in Eq. (21).

$s_{i j}(t)=\frac{\left\langle F_i^{\{ {fusion }\}}(t), F_j^{\{ {fusion }\}}(t)\right\rangle}{\left\|F_i^{\{ {fusion }\}}(t)\right\|\left\|F_j^{\{ {fusion }\}(t)}(t)\right\|}$               (21)

Here, Cosine or Jaccard similarity based on fused features at time t. When two nodes exhibit similar multimodal characteristics, the similarity is high. Then, we have the static spatial node proximity combined with the dynamic similarity as shown in Eq. (22).

$w_{i, j}(t)=\lambda w_{i j}^0+(1-\lambda) \cdot {ReLU}\left(s_{i j}(t)\right)$               (22)

Here, $\lambda \in[0,1]$ a trade-off parameter, ReLU is used to ensure that weights are non-negative so edges remain valid. Then finally, we add temporal continuity by connecting each node across time periods as $\left(v_i(t), v_i(t-\tau)\right) \in E_t, \forall \tau \in \left\{1, \ldots, T_{{lag }}\right\}$. At time $t$, the adjacency matrix is constructed as Eq. (23)

$A(t)=\left[w_{i j}(t)\right]_{i, j=1}^N, W_t=A(t) \cup\{T E\}$               (23)

The properties of the graph evolve over time, accounting for some combination of geometric locality, feature similarity, and temporal continuity.

3.4 Graph Neural Network Module

The GNNM provides the central component of the proposed framework in order to facilitate structured reasoning over the spatiotemporal graph constructed from the multimodal waste data. After the AMFA step and Graph Construction step, each node $v_{i(t)}$ in spatiotemporal graph $G_t=\left(V, E_t, W_t\right)$ is assigned a fused feature embedding $F_i^{ {fusion }}(t)$. The GNNM enables the propagation of information across nodes to edges to help learn context-aware and spatiotemporally consistent representations that are key to accurate waste classification and prediction. Within each GNN layer l, node features are updated by aggregating neighbor information as shown in Eq. (24).

$H_i^{l+1}(t)=\sigma\left(\sum_{j \in N_i(t)} \frac{w_{i j}(t)}{\sum_{j \in N_i(t)} w_{i k}(t)} W^l H_j^l(t)\right)$             (24)

Here, $N_i(t)$ is a representation of the set of neighbors (spatial + temporal), $w_{i j}(t)$ is the dynamic edge weights from graph construction, $W^l$ is a learnable transformation matrix, and $\sigma(\cdot)$ is a nonlinear activation function. The weighted aggregation allows each node representation to encode not only its own features but also some contextual information from spatial and temporal neighbours. In order to increase the model’s adaptability even more as shown in Eqs. (25) and (26).

$\alpha_{i j}^l(t)=\frac{\exp \left({LeakyReLU}\left(a^{\top}\left[W^l H_i^l(t) \| W^l H_j^l(t)\right]\right)\right)}{\sum_{j \in N_i(t)} \exp \left({LeakyReLU}\left(a^{\top}\left[W^{(l)} H_i^{(l)}(t) \| W^{(l)} H_k^{(l)}(t)\right]\right)\right)}$                (25)

$\mathrm{H}_{\mathrm{i}}^{1+1}(\mathrm{t})=\sigma\left(\sum_{\mathrm{j} \in \mathrm{Ni}(\mathrm{t})} \alpha_{\mathrm{ii}}^1(\mathrm{t}) \mathrm{W}^1 \mathrm{H}_{\mathrm{j}}^1(\mathrm{t})\right)$             (26)

Here, $\alpha_{i j}^l(t)$ indicates the importance of neighbor $j$ to node $i$ at layer $l$. Furthermore, temporal self-edges from graph construction allowed for the GNNM to account for short-term and long-term temporal patterns:

$H_i^{l+1}(t)=f\left(H_i^l(t), H_i^l(t-\tau),\left\{H_j^l(t)\right\}_{j \in N i(t)}\right)$              (29)

Here, $f(\cdot)$ notation refers to the combined aggregation of spatial neighbours and temporal history. The ultimate node embeddings $H_i^L(t)$ after $L$ GNN layers are passed to fully connected layers or SoftMax classifiers for downstream tasks.

$\hat{y}_{i(t)}={Softmax}\left(w_{ {out }} H_i^{(L)}(t)+b_{{out }}\right)$               (30)

Here, $y^i(t)$ indicates the predicted waste category or status. This output could also inform energy-aware routing decisions or a dynamic waste management plan in IoT-enabled smart cities.