Enhanced Cigarette Pack Counting via Image Enhancement Techniques and Advanced SAFECount Methodology

The evolution of AI technology has ushered in a new era of applications and advancements, particularly in the domain of counting. Deep learning methods for counting have been effectively utilized across various fields. In urban areas, these methods have been instrumental for crowd monitoring and counting, as observed in busy streets, stations, and stadiums [1-6]. Such applications provide critical insights into crowd dynamics and behaviors. In agriculture, deep learning has been employed for crop quantity estimation, facilitating precision in agricultural management [7-12].

The realm of cigarette pack counting has not been exempt from the influence of these advanced methodologies. Accurate counting of cigarette packs is a pivotal component of industrial production, ensuring the efficiency of production processes. The autonomous nature of deep learning methods enables automated counting in high-activity production environments, thereby obviating the need for manual counting. This automation significantly augments both accuracy and efficiency. Furthermore, these methods exhibit remarkable adaptability in complex scenarios, such as dealing with stacked or obscured cigarette boxes, thereby bolstering production line management.

In the broader context of image-based counting, deep neural networks have become a cornerstone for object counting. Predominantly, these methods are categorized into two types [13]: detection and segmentation-based counting methods, and density estimation-based counting methods. Detection and segmentation-based methods incorporate object detectors and segmentation modules within counting systems. These methods identify objects in an image and proceed to count them, with Mask-RCNN [14] and RetinaNet [15] being notable examples. While offering high accuracy and comprehensive information, these methods are often hampered by higher computational demands and sensitivity to variations in target objects. Conversely, density estimation-based methods approach counting as a regression task, generating a heat map correlated to the image to estimate target object quantity, followed by count regression from the heatmap. LC-DenseFCN [16] and CentroidNet [17] exemplify this category. Characterized by simpler structures and higher computational efficiency, these methods demonstrate greater adaptability to changes in target objects compared to their detection and segmentation-based counterparts. However, applying these methods to cigarette pack counting in warehouse environments presents unique challenges. Firstly, images from cigarette warehouses often suffer from noise interference due to suboptimal lighting conditions or sensor noise. Secondly, motion blur may occur due to vehicle movement during loading processes. The variable appearance of cigarette packs, influenced by different batches or design changes, further complicates the counting task. Moreover, conventional methods, typically trained on fixed datasets, struggle to adapt to evolving cigarette pack designs or promotional variations, leading to diminished counting performance when applied directly to real-world cigarette pack counting scenarios.

To confront these challenges, a plethora of researchers have embarked on refining counting methodologies. A prevalent strategy, data enhancement, has been extensively employed to mitigate image noise and blur [18-25]. In this vein, Zhang et al. [26] introduced a novel approach, Noise2Noise, which interprets annotations as noise. This method establishes a self-supervised pre-training task, employing image enhancement within the Noise2Noise framework. Additionally, Waqas Zamir et al. [27] developed the Restormer model, a robust solution for image denoising. This model integrates a Multi-DCONV Transposed Attention (MDTA) module with a Gate DCONV Feed-forward Network (GDFN), thereby accentuating local spatial contexts and implicitly modeling global pixel relationships. In response to the variable appearances of tobacco packs, the FSC method has gained traction [28-32]. This innovative approach enables users to identify target objects using merely one or a few support images. Significantly, this method does not necessitate an extensive array of training samples for the test objects. Providing the support images are available, a proficiently trained model can adeptly conduct inference on new classes. Within this context, Lu et al. [29] proposed the Graph Matching Network (GMN) model. This model synergizes support features with query features, followed by the learning of regression heads for point-wise feature comparison. However, this approach's comparative effectiveness is relatively diminished when contrasted with using similarity for comparison. Yang et al. [31] introduced the CFOCNet (Coupled Feature-Offset Comparison Network) model, which initiates feature comparison with point generation, subsequently regressing density maps from the derived similarity maps. Ranjan et al. [30] developed the FamNet (Few-shot Adaptive Metric Network) model, which augments the reliability of similarity maps through multi-scale augmentation and test-time adaptation. However, the informational capacity of these similarity maps is somewhat limited. Consequently, regression from these maps may not yield optimal results, particularly in scenarios where objects are densely arranged. Numerous studies have endeavored to enhance general methods from various angles, aiming to tailor them more effectively to the task of cigarette pack counting in real-world settings. Despite these efforts, a comprehensive solution framework that effectively addresses all the aforementioned challenges in their entirety remains elusive.

To address the challenges identified, an innovative two-step solution is proposed. Initially, original images undergo processing through an image enhancement module. Subsequently, an enhanced version of the SAFECount method [32] is employed for the counting process. The first challenge, involving denoising and deblurring, is tackled through the application of advanced image enhancement techniques. In dealing with the frequent variations in the appearance of cigarette packs, it is suggested to amalgamate specialized color and boundary feature extraction networks with the SAFECount method. This combination is key to effectively managing these variations, thereby improving the accuracy of the counting process. To overcome the scalability issues inherent in general counting models, the adoption of a FSC approach is proposed. This method facilitates rapid learning with a minimal number of samples, enabling the model to accurately predict unknown classes based on the knowledge of known classes. This approach significantly enhances the model's generalization and scalability capabilities. The selected representative method for this purpose is the improved SAFECount method. This method incorporates a multi-scale feature aggregation mechanism, enhancing its flexibility and adaptability. By updating and adjusting the model with a small number of support samples, it can swiftly adapt to changes in the content of cigarette packs and perform accurate counting. Empirical studies conducted using the Zhongyan cigarette pack dataset reveal that the proposed method excels in practical engineering applications. It demonstrates superior counting accuracy and scalability compared to existing methods. These experimental findings corroborate the efficacy and reliability of the proposed approach in addressing the challenges of cigarette pack counting tasks.

The contributions of this research are multifaceted and significant:

(1) Image Enhancement: Central to this study is the development of an image enhancement module, comprising two key components: denoising and deblurring. The denoising component substantially reduces image noise, thereby augmenting clarity and detail visibility. Concurrently, the deblurring component addresses image blur, a consequence of camera or object movement, resulting in significantly clearer images. These enhancements collectively bolster object detection capabilities and consequently, improve counting accuracy.

(2) FSC: A novel FSC approach is introduced, enabling swift learning and generalization with a limited number of samples. This methodology allows for efficient adaptation to various tasks and domains. It offers remarkable flexibility and adaptability, significantly reducing manpower requirements and enhancing learning efficiency.

(3) Enhanced SAFECount Method: This research extends the SAFECount method by integrating color semantic information and boundary semantic information into the model. This integration markedly improves the model's detection efficiency and accuracy, positioning it at the forefront of counting technologies.

(4) End-to-End Optimization Approach: A unique aspect of this study is the development of an end-to-end optimization approach. This approach aligns the objectives of image enhancement and counting, facilitating a cohesive and synergistic improvement in detection performance.

2.png

Figure 2. Schematic diagram of cigarette pack counting based on various image enhancement techniques and improved SAFECount

3.png

Figure 3. Schematic diagram of the denoising module

4.png

Figure 4. Schematic diagram of the deblurring module

The final stage involves image reconstruction, employing a residual network augmented with a feature attention mechanism, which is generated through convolutional processes. This mechanism is instrumental in automatically identifying and focusing on pivotal image features, thereby diminishing the impact of noise and augmenting the quality of the denoising results. Crucially, this is achieved while maintaining the integrity of image details.

The meticulously structured network design is thus proficient in accomplishing blind denoising and enhancement of cigarette pack images. It is characterized by a reduced parameter count and efficient computational capacity, rendering it suitable for real-time denoising applications. This innovative approach represents a significant advancement in the field of image processing, particularly in contexts demanding high-quality denoising under stringent computational constraints, as shown in Figure 4.

The second one is the deblurring module. This module is inspired by a successful network structure frequently utilized in image deblurring, known as the image pyramid structure. Utilizing this structure, the feature extraction layers progressively restore a clear image, thereby achieving blind image deblurring. The architecture of this network adheres to a top-down encoder-decoder model, wherein the reconstruction of the clear image is progressively realized by integrating outputs from different scales of network layers. Comprising the network are two sub-networks, each mirroring the same structural composition: seven convolutional layers, two fully connected layers, and one Long Short-Term Memory (LSTM) layer. The process initiates with feature extraction executed through convolutional layers, followed by image restoration via deconvolutional layers. A "coarse-to-fine" strategy is central to this process. This approach involves a progressive reduction in the size of the source image, with the clear image being restored based on inputs at varying resolutions:

$I_i, h_i=\operatorname{Net}_i\left(B_i, I_{i+1} \uparrow, h_{i+1} \uparrow\right)$                (1)

where, i denotes the resolution index, i=1 corresponds to the original image resolution. B_iand C_i represent the blurry image and the restored clear image at the i-th resolution, respectively. Net_i signifies the i-th layer sub-network, while h_i indicates hidden features. The upsampling of parameters to match the network layers is denoted by the operator. The transformation of the input image into feature maps involves a reduction in dimensions but an increase in the number of channels, facilitated by a CNN network. To capture hidden features from low-resolution inputs, an LSTM network is integrated, thereby enhancing the expressive capacity of the feature maps. Additionally, an attention mechanism is implemented, dynamically adjusting the weights of the feature maps. This adjustment is responsive to variations in image content, further improving the deblurring effects. This sophisticated approach, combining the strengths of the image pyramid structure, LSTM networks, and attention mechanisms, significantly advances the capability of image deblurring, particularly in challenging scenarios such as those presented by cigarette pack images.

3.3 Feature extractor

The feature extraction network consists of two specialized subnetworks: one for color feature extraction and another for edge feature extraction. Support images and query images are processed through these subnetworks to extract pertinent features. The color feature extraction network isolates features rich in color information from the support images, while the edge feature extraction network focuses on extracting boundary-related features. These extracted features are then fused with the query features using an additive fusion technique. This process entails the pixel-wise addition of corresponding positions in the feature maps of both the support and query features. The resultant combined features, encompassing both color and boundary information, are subsequently utilized as inputs for the counting module, illustrated in Figure 5.

5.png

Figure 5. Schematic diagram of feature extraction

In the Color Feature Extraction process, feature extraction is executed in two distinct steps. The first step involves initially extracting color features through a convolutional network, while the second step utilizes self-attention mechanisms to enhance the focus on color features, thereby improving cigarette pack recognition and overall performance, detailed in Figure 6.

During the first step, the input image is processed through two convolutional layers, followed by a ReLU activation function and a max-pooling layer, to extract low-level features. A deconvolutional layer is then employed to upscale the low-level feature map back to the original input image size. These low-level features undergo normalization using a SoftMax function, ensuring the feature values are confined within a specific range. The normalized low-level features are further processed through color mapping via a fully connected layer, aligning the feature values with the color space. The resultant mapped feature map is then element-wise multiplied with the original image, infusing color information into the image.

In the second step, the color-weighted image from the first step is further refined. Initially, it is passed through two convolutional layers to generate a feature map rich in color semantic information. This is followed by the application of three 3x3 convolutional kernels with ReLU activation functions, mapping the input feature map into spatial representations for Query, Key, and Value features. Point-wise multiplication between the Query and Key features generates a matrix, indicative of the similarity between queries and keys. This similarity matrix undergoes a Softmax operation to derive attention weights, where each element reflects the relative importance of its position. Finally, these attention weights are matrix-multiplied with the Value features, producing a feature representation attentively weighted at each position. This methodical approach ensures a detailed and focused enhancement of color features, crucial for the accurate recognition of cigarette packs.

6.png

Figure 6. Schematic diagram of the color feature extraction

7.png

Figure 7. Schematic diagram of the boundary feature extraction

8.png

Figure 8. Schematic diagram of the counting block

To realize this, a SCM and a FEM are proposed, as depicted in the accompanying figure. The SCM extends beyond basic feature comparisons by learning feature projections and executing comparisons on these projected features to derive a score map. This process is formulated as follows:

$\boldsymbol{R}_{\mathbf{0}}=\operatorname{conv}\left(g\left(\boldsymbol{f}_{\boldsymbol{Q}}\right)\right.$, kernel $\left.=g\left(\boldsymbol{f}_{\boldsymbol{S}}\right)\right)$                 (2)

where, $\boldsymbol{R}_{\mathbf{0}}$ represents the score map, $\boldsymbol{f_Q}$  stands for the query feature, $\boldsymbol{f_S}$ represents the support feature, and $g()$ denotes the feature projection, achieved through a 1×1 convolutional layer followed by layer normalization. This design aids in selecting the most relevant information for object counting from the features. After the comparison step, the score maps associated with all support images (few samples) are aggregated. This aggregation is followed by normalization along two dimensions: the exemplar dimension and the spatial dimension. The normalization process is essential for generating a reliable similarity map, which is instrumental in the counting process. The computation of the similarity map is defined by the following formula:

$\boldsymbol{R}=\boldsymbol{R_{E N}} \otimes \boldsymbol{R_{S N}}$            (3)

where, $\boldsymbol{R}$ represents the similarity map, $\boldsymbol{R}_{\boldsymbol{E N}}$ denotes the exemplar normalization result, and $\boldsymbol{R}_{\boldsymbol{S N}}$ signifies the spatial normalization result. The formula for the exemplar normalization result, $\boldsymbol{R_{E N}}$, is as follows:

$\boldsymbol{R}_{\boldsymbol{E N}}=\operatorname{softmax} {dim}=0\left(\frac{\boldsymbol{R}_{\mathbf{0}}}{\sqrt{H_S W_S C}}\right)$              (4)

The formula for the spatial normalization result, $\boldsymbol{R_{S N}}$, is as follows:

$\boldsymbol{R}_{\boldsymbol{S N}}=\frac{\exp \left(\frac{\boldsymbol{R}_{\mathbf{0}}}{\sqrt{\left(H_S W_S C\right)}}\right)}{\max _{\operatorname{dim}=(2,3)}\left(\exp \left(\frac{\boldsymbol{R}_{\mathbf{0}}}{\sqrt{H_S W_S C}}\right)\right)}$                   (5)

The $max _{\operatorname{dim}=0}$ function in the equation represents finding the maximum value along the specified dimension. Furthermore, FEM utilizes point-to-point similarity as weighting coefficients to fuse support features into query features. The implementation steps are:

In the first step, aggregation of $\boldsymbol{f}_{\boldsymbol{S}}$ into $\boldsymbol{f}_{\boldsymbol{R}}^{\prime}$ is achieved through the similarity $\boldsymbol{R}$:

$\boldsymbol{f}_{\boldsymbol{R}}^{\prime}=\operatorname{conv}\left(\boldsymbol{R}\right.$, kernel $\left.=\operatorname{flip}\left(\boldsymbol{f_S}\right)\right)$                  (6)

In the second step, the sum $_{\text {dim }}$ function is employed to accumulate $\boldsymbol{f}_{R}^{\prime}$ along a specific dimension, resulting in the similarity-weighted feature $\boldsymbol{f_R}$:

$\boldsymbol{f_R}=\operatorname{sum}_{\operatorname{dim}=0}\left(\boldsymbol{f_R^{\prime}}\right)$              (7)

Finally, $\boldsymbol{f}_{\boldsymbol{R}}$ is efficiently fused into $\boldsymbol{f}_{\boldsymbol{Q}}$ through a network, yielding the enhanced feature $\boldsymbol{f}_{\boldsymbol{Q}}^{\prime}$:

$\boldsymbol{f}_{\boldsymbol{Q}}^{\prime}=$ layer_norm $\left(\boldsymbol{f}_{\boldsymbol{Q}}+h\left(\boldsymbol{f}_{\boldsymbol{R}}\right)\right)$                   (8)

The fusion process within the SAFECount method ensures that the enhanced query feature intensifies its focus on areas resembling the exemplar objects as defined by the support images, thus facilitating more accurate counting. The integration of the SCM and FEM allows for the optimal utilization of similarity information to refine the accuracy of feature representation. Consequently, the enhanced features are enriched with more profound semantic information and are adept at prioritizing areas analogous to exemplar objects, guided by the support images. This leads to a notable improvement in counting precision.

In this study, the number of SAFECount blocks employed is three. Experiments were conducted to determine the optimal number of basic blocks. It was observed that as the number of SAFECount blocks increased from one to three, there was a corresponding enhancement in the model's technical performance. However, upon increasing the number of blocks to four, a decline in performance was noted. This decrease in performance can be attributed to the model becoming excessively complex with the additional blocks, leading to overfitting issues. The specific outcomes of these experiments are detailed in Table 1.

Table 1. Number of SAFECount block

An analysis of Table 4 reveals that, across all dataset splits and irrespective of the number of support images used, the proposed model consistently outperforms FamNet. In the 1-shot scenario, our model demonstrates an average improvement of 1.18 in MAE and 3.41 in RMSE when compared to FamNet. In the more challenging 3-shot scenario, the proposed model shows an even more pronounced improvement, with a 1.44 increase in MAE and a 2.81 increase in RMSE relative to FamNet. These results suggest that the proposed model is more adept at learning the similarity relationship between support and query features, thereby yielding superior counting accuracy.

When comparing the 1-shot and 3-shot scenarios as outlined in the table, it is apparent that the proposed model attains more significant performance gains compared to the FamNet method. In the case of our model, the 3-shot scenario shows a 32% enhancement in MAE and a 57% improvement in RMSE over the 1-shot scenario. Conversely, FamNet exhibits only a 14% increase in MAE and a 22% increase in RMSE in the 3-shot scenario compared to the 1-shot scenario. This disparity highlights the effectiveness of the proposed model in leveraging the advantages of multiple support images. The ability to utilize additional support images more efficiently translates into notably improved performance in practical cigarette pack counting tasks, emphasizing the robustness and adaptability of the proposed counting approach.

5.5 Cross-validation experiments

To substantiate the enhancements brought by the image enhancement module, along with the color and edge feature extraction modules in addressing cigarette pack counting challenges, ablation experiments were conducted on various components of our proposed approach. These included the denoising module, brightness enhancement module, deblurring module, and feature extraction module. The objective was to demonstrate the individual and collective impact of each module on the final detection outcomes. The results of these experiments are summarized in the accompanying table. "Baseline" in this context refers to the SAFECount model devoid of these enhancement modules. "M1,""M2," and "M3" correspond to the inclusion of the denoising module, deblurring module, and our specifically developed feature extraction module, respectively. "M4" indicates the combined application of the denoising and deblurring modules, "M5" denotes the integration of the denoising module with the feature extraction module, "M6" represents the combination of the deblurring module and the feature extraction module, and "M7" encapsulates the complete methodology as proposed in this paper.

Table 5. The test results of the ablation experiments conducted on the Zhongyan cigarette pack dataset using our proposed method

The analysis of Table 5 reveals notable improvements in performance metrics when comparing various model configurations with the baseline SAFECount model. M1, which incorporates the denoising module, demonstrates superior performance over the baseline, indicating its efficacy in enhancing the model's recognition of cigarette packs in noisy environments. This is quantified by a 4.7% increase in MAE and a 9.1% increase in RMSE.

Similarly, M2, which includes the deblurring module, effectively counters the effects of image blurriness, leading to a 3.2% rise in MAE and a 7.1% rise in RMSE compared to the baseline. This underscores the module's contribution to improving the model's technical performance in complex environments.

The comparison between M3 and the baseline further demonstrates the effectiveness of the proposed feature extraction module, particularly in extracting individual cigarette packs in complex scenarios, resulting in a 2.1% improvement in MAE and a 4.6% improvement in RMSE.

When examining M4, which combines the denoising and deblurring modules, there is a more significant improvement across all performance metrics, with increases of 5.8% in MAE and 11.6% in RMSE over the baseline. Likewise, M5, which merges the denoising module with the feature extraction module, shows enhanced improvements, with a 5.8% rise in MAE and a 10.8% rise in RMSE. M6, combining the deblurring module and the feature extraction module, also shows superior performance, with a 5.3% increase in MAE and a 9.1% increase in RMSE.

Notably, the results of M4, M5, and M6 surpass those of M1, M2, or M3 when used individually, suggesting that the combined usage of these two modules can further boost the model's performance.

Finally, M7, which represents the full integration of the three modules as proposed in this paper, shows the most significant improvements. The MAE and RMSE values increase by 10.0% and 19.1%, respectively, compared to the baseline. This comprehensive integration demonstrates that each module contributes uniquely and synergistically, greatly enhancing the overall model's performance in the precise task of cigarette pack counting.