Enhancing Pharmacy Warehouse Management With Faster R-CNN for Accurate and Reliable Pharmaceutical Product Identification and Counting

Abstract

The rise of digitalization and Industry 4.0 has led to significant changes in industrial warehouse management. However, managing warehouses remains challenging due to reliance on manual labor and limited automation. This article focuses on addressing issues in warehouse management, specifically in drug identification and counting. Although traditional methods such as barcode systems and RFID are common, artificial intelligence (AI) offers a promising solution. In this paper, an advanced visual recognition based on Faster R-CNN is introduced to accurately identify and count pharmaceutical items in pharmacies. The obtained results suggest that intelligent warehouse management in pharmacies can lead to cost savings and improved efficiency. The study also compares the proposed model with popular classification methods such as CNN, SVM, KNN, YOLOv5, and SSD, showing the effectiveness of the new approach.

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

1. Introduction

In the past decade, the world has witnessed a rapid shift toward digitalization and the adoption of Industry 4.0 technologies. However, certain sectors within the manufacturing industry still face significant challenges regarding automation and the integration of digital solutions. In the field of warehouse automation, challenges frequently arise during the design and implementation phases. Primary activities in warehouse management encompass product storage, risk management, financing, grading, branding, and transportation [1]. In many warehouses, these tasks are carried out manually or with limited automation [2, 3]. This manual operation leaves room for human error, which is the primary cause of data management mistakes [4]. Data accuracy within the warehouse system module is crucial for effective production planning and supply chain management.

Nowadays, companies are experiencing an increase in trade volume and the complexity of their supply chain networks is growing. Therefore, their warehouses consist of connected storage units and racks, creating a network [5]. It can show the importance of warehouse management that can play a vital role in enhancing industrial productivity and providing better customer service [6]. Accurate warehouse information ensures smoother production processes and timely delivery of commercial goods to customers [7]. On the other hand, human mistakes can lead to significant losses for companies during practical operations. As companies experience an increase in trade volume, the complexity of their supply chain networks also grows. Warehouses consist of connected storage units and racks, creating a network [7]. Managing and maintaining such a complex network is challenging, and the task of monitoring and tracking all the goods within it becomes even more laborious. There is an urgent need to effectively oversee all ongoing processes within a warehouse and create a dynamic storage facility that facilitates easy accessibility [8, 9]. As a critical and fundamental component of the supply chain, warehouse management has always been an important area of study for researchers [10–14]. However, one crucial function of warehouse management, namely, components receiving [15], still requires human involvement. This includes checking purchase orders, counting parts, and inputting the data into the enterprise system. These receiving data form the foundation for all supply chain management and industry operations. These issues, along with balancing the need to have adequate stock without overstocking, are the most challenging problems for a pharmacy. Pharmacies are one of the most important parts of the human and animal healthcare and well-being system in all countries. Efficient storage, management, and distribution of pharmaceutical products play an important role in the operations of a pharmacy. Accurate and reliable warehousing helps in preserving the quality and safety of medications. Using artificial intelligence (AI) and machine learning (ML) can solve current challenges in the industry especially for pharmacies, improving productivity and efficiency in warehouses. While AI has gained substantial traction in various engineering fields, its utilization as a valuable problem-solving tool in warehouse management still requires further exploration [16, 17]. In the field of supply chain management, AI has been gradually adopted for sales data analysis, customer demand analysis, and the distribution of warehouse production orders [18–20]. Several models have been developed to classify and identify stock levels [6, 21, 22]. Counting objects by their classification is still a difficult problem in warehouse management, and only a few studies have explored this topic. In contrast, AI has made substantial strides in object recognition, making it one of the most accurate branches of AI, with extensive research conducted in this field. Thus, utilizing AI to identify, count, and manage inventory can be an efficient and accurate tool. Presently, AI capabilities primarily revolve around prediction, planning, and learning. Importantly, these AI capabilities are intertwined with the overarching objectives of warehousing. These AI capabilities are closely linked to the main goals of warehousing. They work with human skills and require constant improvements to create business value through increased efficiency (e.g., space optimization and workforce productivity) and effectiveness (e.g., error reduction) [23].

Counting and identifying items within a warehouse or in available shelves for a pharmacy, followed by efficient warehouse management, are critical measures in pharmaceutical warehouse management. While numerous technologies (e.g., RFID [24] and QR code [25]) have been developed for item identification, they often require more continuous monitoring capabilities. Implementing AI tools for item counting in pharmacy warehouses can significantly reduce costs, enhance speed and efficiency in warehouse management, and ultimately lead to increased profitability for pharmacy owners [26, 27]. Moreover, the application of these advanced technologies brings additional benefits such as improved safety measures, decreased incidents of theft and pilferage, real-time inventory tracking capabilities, and optimization of warehouse operations [28]. Our study has extensively examined the important area of inventory control in pharmaceutical warehouses, with a particular focus on two key aspects: drug identification and counting. In the following section, we explored the methodologies that contribute to pharmacy warehouse management, which can be classified into two main categories.

1.1. Warehouse Management Without AI

A pharmacy barcode system is a technology that uses barcodes to track and manage medication inventory in a pharmacy. Each medication is assigned a unique barcode that can be scanned using a scanner. The system can track the medication from the time the pharmacy receives it until it is given to a patient. This helps to ensure that the right medication is given to the right patient, which reduces the risk of medication errors and helps to manage inventory levels. In addition, the system can generate reports on medication usage, expiration dates, and other important information to help pharmacy staff make informed decisions about inventory management. RFID technology has gained significant popularity in product segmentation, tracking, and positioning [29, 30]. The information collected by the RFID reader is transmitted wirelessly to open-source hardware, and the resulting data are stored in a central server. All modules are interconnected through a common network to ensure efficient data communication. The authors in [30] proposed an IoT-based WMS for industries that specialize in developing customized products. The system utilizes RFID tags to label all parts, which are assembled to create the final product. The RFID information collected from these parts is transmitted to an Electronic Product Code (EPC) information server, responsible for monitoring inventory levels, making modifications, handling change requests, and picking faults. It is a common practice to use vehicles for transportation within a warehouse. In the past, manual vehicles were common, but now IoT uncrewed vehicles, robots, and conveyor belts are becoming more popular. The authors in [31] created a system that uses unmanned aerial vehicles (UAVs) to manage warehouse inventory. These UAVs can count and find inventory while avoiding obstacles and collisions.

1.2. Warehouse Management System Using AI

A literature search shows no published studies that explicitly explore the application of AI in pharmacy warehousing. However, over the past decade, AI has emerged as a significant contributor to the field of supply chain management. Research efforts have primarily concentrated on utilizing AI to predict customer demands, fulfill orders, and optimize component pick-up from designated locations within the warehouse [22, 32–34]. AI technologies have also found widespread applications in security and business domains, including warehouse management [35–40]. Several recent studies have introduced models for the classification and counting of various objects, including individuals, cars, and other items [35, 41–46]. These studies offer valuable references for developing neural network models in the context of industrial component identification and counting research. There exist several well-known convolutional neural network (CNN) applications, including AlexNet [40] and GoogLeNet [42], which have been proposed as sufficient for image classification. These networks are designed to recognize objects in images and are suitable for industrial component classification and counting in warehouse management. This section introduces five standard neural network model structures, such as AlexNet, VGG [47, 48], GoogLeNet, and SSD [49], which can be tailored to meet specific design requirements by adjusting layers and fine-tuning outputs. However, there are still challenges in adding these models to industrial warehouse management systems, especially when it comes to ensuring accurate predictions. CNN architectures are known for their higher prediction accuracy compared to simpler neural networks. Studies have shown accuracy rates of over 96% on datasets such as MNIST and Human Face datasets, according to Müller, Guido, and Geron [50, 51].

In [52], three object detection methods Faster R-CNN, YOLO, and SSD are evaluated for their effectiveness in identification tasks. The study highlights that Faster R-CNN is particularly well-suited for identification compared to YOLO and SSD for several reasons: Higher mean average precision (MAP): Faster R-CNN achieves a MAP of 87.69%, significantly exceeding the performance of YOLOv3 (80.17%) and SSD (82.41%). This superior precision (P) is crucial in pharmacy environments, as accurate drug identification is vital for minimizing medication errors and enhancing patient safety. Emphasis on accuracy: While its detection speed (7 FPS) may be slower than the alternatives, Faster R-CNN prioritizes accuracy over speed. This focus ensures that pharmacists can rely on the results, thereby reducing the likelihood of misdispensing medications in critical situations.

As outlined in this section, various approaches can be suggested for managing pharmacy warehouse management systems. Nevertheless, concerning our objective in this article, which pertains to drug identification and counting, none of the methods, except for AI, can effectively accomplish this goal, because the alternative methods, such as the pharmacy barcode system and RFID technology, are time-consuming and costly, in contrast to warehouse management system using AI. Using AI tools for counting the items in the pharmacy warehouse can reduce costs and increase the speed and efficiency of managing a warehouse with the development of CNN [53–55].

In the evolving landscape of warehouse management, especially in the pharmaceutical sector, there are several unique challenges that need academic focus. Even with advances in automation and the quick use of AI technologies, many warehouses still depend on traditional manual processes. This reliance not only amplifies the risk of human error in data management but also impedes the efficiency required for contemporary supply chains. One of the primary challenges is ensuring data accuracy within warehouse management systems, which is vital for effective production planning and supply chain operations. Pharmaceutical inventory control is very detailed, requiring accurate records of drug identification and quantities, which highlights the problems caused by incorrect data. In addition, the complex structure of warehouse networks creates more challenges, making strong monitoring and tracking systems necessary, something that current manual systems do not provide. Another significant challenge is fitting AI and ML into current warehouse management practices. Although there have been major advancements in other engineering fields, the use of AI for object classification and counting in warehouse management is still not well studied. Traditional technologies, such as barcode scanning and RFID systems, often fall short in facilitating the real-time accuracy and efficiency that AI tools can offer. The gap between current methodologies and the potential of AI technologies highlights an urgent need for innovative solutions that can enhance the operational efficiency and accuracy of pharmaceutical warehouse management. The expensive and slow nature of traditional methods for drug identification and counting highlights the need for better solutions that use AI. Using AI-driven systems could greatly help with inventory management by providing timely data that are crucial for making informed decisions. In addition, while current systems use neural networks for classification and recognition, they face major challenges when being integrated into real warehouse management systems, especially in achieving high prediction accuracy. Lastly, combining AI capabilities with human work presents a complex challenge that needs more research. Ensuring that AI systems work smoothly with human efforts to boost productivity requires continuous study of their design and implementation. By tackling these various challenges, the research community can improve warehouse management practices and create a stronger, more efficient supply chain. We have successfully designed and implemented a sophisticated AI-driven warehouse management system tailored specifically for pharmacies. This system addresses the traditional challenges faced in warehouse management by utilizing advanced AI techniques to enhance operational efficiency. The research introduces a novel application of visual recognition methods, specifically employing the Faster R-CNN algorithm [38, 56, 57], to accurately identify and quantify pharmaceutical items. This approach not only streamlines the inventory management process but also significantly reduces human error associated with manual counting. The proposed system integrates AI capabilities that facilitate real-time tracking and management of pharmaceutical inventory. It optimizes drug identification and counting processes, addressing critical issues present in traditional methods such as barcode systems and RFID technology, which often lack the P and speed required for efficient warehouse operations. Practical applications for pharmacy owners: Beyond theoretical contributions, the developed system is designed with practical applications in mind, aiming to provide pharmacy owners with an application that not only meets their operational needs but also enriches their service delivery and customer satisfaction.

2. Proposed Method

One of the main challenges in warehousing for pharmacies is the lack of enough datasets for developing reliable AI-based models. Therefore, the first step is to collect a diverse range of drug samples to create a dataset of drug images. Once the dataset is collected, it should be labeled appropriately. Each image in the dataset should be annotated with the corresponding drug name or class. The second challenge is to develop an accurate AI-based model for identifying and counting the drugs in the images. In this section, we first discuss how to create a dataset of drug images, and then the structure of the proposed algorithm for identifying and counting drugs.

2.1. Collecting and Generating Dataset

Having a reliable image database is crucial in image processing systems that aim to identify and count objects. A well-organized image database can provide the necessary reference images for comparison and analysis, allowing the system to identify and count objects in new images accurately. A comprehensive image database can also improve the system’s ability to recognize and classify objects based on their features and characteristics. Therefore, the importance of a robust image database must be considered in image processing systems that require object identification and counting.

2.1.1. Classification of Pharmacy Items

In this section, we aim to explain how a comprehensive image database in the pharmaceutical field has been collected. To ensure accuracy and reliability, we thoroughly reviewed all items in the pharmacy database, considering the significant scatter in the data. In addition, we carefully selected a specific number of items from each category to ensure a balanced representation of the database’s centralized data. All pharmaceutical products can be categorized into 4 different groups. Table 1 lists the categories of items and the specifications of drugs in each category, and examples of these items are shown in Figure 1.

Table 1

Classification of pharmacy items.

Category	Description	Specifications	Sample of products
1	Cardboard or plastic rectangular cube	Rectangular cube cardboard packaging	Boxes of all kinds of pills and capsules
2	Packaging with plastic, metal, glass, and cardboard bottles and cans	Cardboard envelopes, glass, and metal cans	Syrup bottle, suspension, dropper, and vial
3	Flexible plastic or cardboard envelope	Flexible plastic envelope shaped like a rectangular cube	All kinds of medical adhesives (wound adhesive, angiocath, and injection), all kinds of pads and bandages, and packaging of all kinds of syringes
4	Flexible plastic envelope with the possibility of vertical arrangement	The difference between these items and the previous category is that they can be arranged vertically according to the place of installation or the volume inside the package, and they can be set in 4 directions.	Packaging of herbs, supplements, and kits

[figure(s) omitted; refer to PDF]

According to Table 1, 40 drugs have been selected to create the initial dataset, and to cover all possible real positions and conditions of image acquisition, five different angles along with two different illumination conditions (light and dark) have been considered as shown in Figure 2 to generate different samples for an individual drug. These five angles collectively encompass the following:

• The angle is 90° horizontally and 0° vertically

• The angle of 45° west vertically and 90° horizontally

• The angle of 45° east vertically and 90° horizontally

• The angle is −45° horizontally and −45° vertically

• The angle is 45° horizontally and 45° vertically

[figure(s) omitted; refer to PDF]

2.1.2. Data Preparation

As mentioned in the previous section, all collected data are 4 × 10 × 5 × 2 = 400 images. The dataset collected from drug images is divided into training, validation, and testing sets. These sets train the algorithm, adjust its performance, and evaluate its accuracy. In the first step, we labeled all the images using the bounding box technique. In the second step, we added augmentations such as different rotations, two light intensities of 150 and 250 lux, blur, overlap drugs, and salt and pepper noise to increase the number of images in the database. After applying the mentioned augmentations, we developed and tested the models by using 6146 images. Figure 3 illustrates some implemented augmentations for a pharmaceutical production that demonstrate the diverse environment and conditions encountered in a real-world pharmacy.

[figure(s) omitted; refer to PDF]

2.2. Proposed AI-Based Model

In this article, a Faster R-CNN authentication system detection algorithm is used. To understand the Faster R-CNN algorithm, we briefly explain the R-CNN and Fast R-CNN algorithms. In 2014, UC Berkeley researchers introduced R-CNN, a deep convolutional network for object recognition [58]. R-CNN utilizes a CNN for feature extraction and consists of three modules: region proposal generation, feature extraction, and support vector machine (SVM)–based classification. However, R-CNN has drawbacks such as multiple stages, storage requirements, and reliance on selective search [55]. To address these issues, Fast R-CNN was developed by Ross Girshick [58]. Fast R-CNN employs ROI pooling to extract feature vectors from regions of interest in an image, resulting in faster processing. Unlike R-CNN, Fast R-CNN shares computations across proposals and requires less disk space [37]. The Faster R-CNN [57] algorithm is an extension of the Fast R-CNN [58] algorithm. Quicker R-CNN is faster than Fast R-CNN, as its name implies, due to the region proposal network (RPN). As shown in Figure 4, Faster R-CNN is composed of three parts as follows.

[figure(s) omitted; refer to PDF]

2.2.1. Convolutional Layers

In these layers, filters are trained to extract the appropriate features of the image. Convolution networks are generally composed of convolution layers, pooling layers, and a last component, which is the fully connected or another extended thing that will be used for an appropriate task such as classification or detection.

2.2.2. RPN

The RPN is a fully convolutional network that creates proposals with different scales and aspect ratios. The RPN applies neural network terminology with attention to instruct the object detection (Fast R-CNN) where to look. Rather than using pyramids of pictures (i.e., multiple instances of the same image at different scales) or pyramids of filters (i.e., multiple filters of varying sizes), this method uses pyramids of pixels. Anchor boxes are reference boxes with specific dimensions and aspect ratios. When multiple reference anchor boxes are used, numerous scales and aspect ratios exist for a single region. This can be viewed as a pyramid of anchor boxes for reference. Each zone is then transferred to its respective reference anchor box, detecting objects with varying sizes and aspect ratios. Both the RPN and the Fast R-CNN share convolutional computations, which decreases the processing time.

2.2.3. Classes and Bounding Boxes’ Prediction

The final part of the Faster R-CNN is to predict object class and bounding boxes. The information is transferred from RPN to another fully connected neural network that works as a classifier and predictor of bounding box points.

2.3. Evaluation Metrics

As shown in Figure 5, the output of the Faster R-CNN model includes the coordinates of various bounding boxes, their corresponding labels, and a score. The score represents the probability of an object belonging to the most relevant class for the given product. To accurately count the objects in an image, a threshold value (τ) is utilized. Only objects with a probability score exceeding the specified threshold are considered valid objects, and the counting process is performed exclusively on objects surpassing the threshold value (τ). One of the desired criteria for identifying objects is IOU that is shown in equation (1). Using this criterion, the accuracy of the RPN network is calculated. The accuracy of the optimal evaluation is evaluated with the community intersection (IOU) metric. In this measure, the overlapping area is divided into the total object areas, and a number smaller than one is considered an IOU.$$\begin{array}{}\text{(1)}& \text{IoU}=\frac{\text{area}\text{of}\text{overlap}}{\text{area}\text{of}\text{union}}\times 100.\end{array}$$

[figure(s) omitted; refer to PDF]

The more significant the overlap of the detected area with the target area, the closer this value will be to 100. We used the F1 measure to evaluate the performance of the trained model. F1 measure is P and recall (R) harmonic mean. It is used when the dataset is not balanced (unequal number of items in each class) as obtained in the testing dataset. F1 measure is calculated using the following equation:$$\begin{array}{}\text{(2)}& {f1}_{\text{score}}=\frac{2\text{PR}}{P+R},\end{array}$$where P and R are given by$$\begin{array}{}\text{(3)}& P=\frac{\text{TP}}{\text{TP}+\text{FP}},\text{(4)}& R=\frac{\text{TP}}{\text{TP}+\text{FN}},\end{array}$$where TP is true positive, FP is false positive, and FN is false negative. Moreover, one of our goals in this article was to count the objects in an image. For this purpose, as mentioned in the previous section, a threshold limit τ was used to count the items in the image, and the accuracy of counting is essential for the correct selection of this threshold limit. Equation (5) has also been used to check the performance of the counting error in an image.$$\begin{array}{}\text{(5)}& \text{Error\hspace{0.17em}for\hspace{0.17em}one\hspace{0.17em}image}=\frac{N-N^{\prime }}{N},\end{array}$$where N is the number of objects of a product in the image and $N^{\prime }$ is the number of identified images. The total error rate for all images equals the average error.

3. Results and Discussion

The simulation used a Python 3 programming environment, PyTorch deep learning library, and a GPU system with 15 GB RAM and seven computing cores. The processing time for the target database was 5 h, 12 min, and 32 s.

3.1. Sensitivity Analysis

To have a robust, accurate and reliable model, we need to use optimum parameters for Faster R-CNN. In this subsection, first, all well-known optimization algorithms have been evaluated, and then the impact of the learning rate on F1 score and the role of batch size have been evaluated. For applying this sensitivity analysis, the size of the training, validation, and testing dataset are 70%, 20%, and 10%, respectively.

3.1.1. Optimizer Algorithm

Selecting a suitable optimizer can have a crucial impact on the model’s performance. Good and compatible optimizers can prevent exploding or vanishing gradients which may lead to disruption and cause instability. In addition, a good optimizer helps the model generalize better to unseen data and reduce overfitting. A proper optimizer can also minimize computational overhead by reducing the number of epochs required to reach an acceptable performance, saving time and computational resources. Four different optimizers including stochastic gradient descent optimize (SGD), Adam, RMSprop, and LBFGS have been considered to find the best one for our case. For this analysis, default values have been considered for learning rate and batch size as 0.001, and 36, respectively. As shown in Figure 6, the SGD optimizer presents better performance. SGD optimizer [59] is widely used to minimize the loss function by iteratively adjusting the model’s parameters based on the gradients of the loss function concerning those parameters. The SGD optimizer updates the model’s parameters in small batches (or randomly selected individual samples) rather than using the entire dataset simultaneously. This stochastic nature allows it to navigate complex and high-dimensional parameter spaces effectively. By iteratively updating the parameters, the SGD optimizer helps the model converge toward an optimal set of parameter values that minimize the loss and improve the model’s performance.

[figure(s) omitted; refer to PDF]

3.1.2. Learning Rate

Another important parameter is the learning rate that can have a direct impact on training procedures. The correct learning rate can balance speed and P, accelerate convergence, and improve overall performance, while high and low rates can increase the risk of underfitting and overfitting, respectively. For this analysis, the optimization algorithm is SGD and batch size is 36. Figure 6 shows the variation of F1 score versus different learning rates. As shown in Figure 7, the F1 score starts to decrease after 0.005, and to avoid underfitting, we selected 0.005 for the learning rate.

[figure(s) omitted; refer to PDF]

3.1.3. Batch Size

Batch size refers to the number of the selected training samples in one iteration to update the weights of the model. Small size can lead to better generalization but potential instability. On the other hand, a large size can lead to more stable and better computational efficiency but increases the risk of overfitting. For this analysis, the optimization algorithm is SGD, learning rate is 0.005, and size of training, validation, and testing data are 70%, 20%, and 10%, respectively. Figure 8 shows that less values for batch size, while optimizer and learning rate can control stability, could be better choices for our applications. Therefore, Batch size 4 has been selected for the developed model.

[figure(s) omitted; refer to PDF]

3.2. Simulation Results

According to the sensitivity analysis, SGD has been selected for optimization and the values for learning rate and batch size are 0.005 and 4, respectively. Figure 9 shows the convergence of lost function for training and validation data that confirms that there is no underfitting or overfitting during the training procedure. Moreover, Figure 10 shows the performance of the proposed model for identifying and counting test images on pharmacy shelves. In these images, the green box is the area prepared in the dataset for the images, and the red box is the area identified by the algorithm. All these obtained results reveal that the proposed AI-based model has a very promising performance on the test data and the error for detecting and counting the items is too low.

[figure(s) omitted; refer to PDF]

3.2.1. Comparison

As mentioned before, there are several traditional AI and ML methods, and deep learning–based models have been applied to warehousing challenges such as product recognition, classification, and inventory management. In this section, the proposed method has been compared with different strategies using three key metrics: accuracy, F1 score, and computational cost and P–R curve.

As can be seen in Table 2, for traditional methods, three well-known features including color features, histograms of oriented gradients (HOGs) [60], and local binary patterns (LBPs) [61] are combined with classifiers such as SVMs in various configurations (quadratic, cubic, linear, and Gaussian) and k-nearest neighbors (KNNs). While these traditional methods have lower computational costs than deep learning models, their accuracy and F1 scores fall short of meeting the practical requirements of modern applications. Experimental results show that although most of them have less computational cost in comparison with deep learning models, their accuracy and F1 scores are less than the desired values. However, information fusion at the feature level can increase the accuracy up to 90% which is more than YOLO and SSD, which is so promising.

Table 2

Comparison of accuracy, training time, and execution time with other methods.

Strategy	Feature	Classifier	Type	Accuracy (%)	F1 score	Training time (m)	Test time (s)
Traditional methods	Color	SVM	Quadratic	79.5	0.77	57.66	1.245e − 3
			Cubic	78.7	0.73	58.085	3.321e − 3
			Linear	77.8	0.64	50.23	2.528e − 3
			Medium Gaussian	71.4	0.59	56.87	2.982e − 3
		KNN	Weighted	70.5	0.55	4.120	2.265e − 5
			Fine	71.9	0.63	3.925	1.892e − 5
	HOG	SVM	Quadratic	82.9	0.77	24.84	2.325e − 4
			Cubic	82.3	0.81	24.81	2.294e − 4
			Linear	71.9	0.69	21.94	2.843e − 4
			Medium Gaussian	77.1	0.67	23.77	3.585e − 4
		KNN	Weighted	73.4	0.55	1.36	1.025e − 5
			Fine	81.3	0.81	1.84	1.125e − 5
	LBP	SVM	Quadratic	84.7	0.84	68.54	5.658e − 3
			Cubic	84.0	0.83	69.04	6.548e − 3
			Linear	79.2	0.74	62.08	5.259e − 3
			Medium Gaussian	78.9	0.77	71.03	8.296e − 3
		KNN	Weighted	73.8	0.70	4.36	2.658e − 5
			Fine	80.8	0.80	4.84	2.952e − 5
	Combined (color, 2 HOG, and LBP)	KNN	Fine	90.24	0.89	12.56	8.386e − 3
Deep learning–based models	CNN			93.25	0.91	41.25	4.595e − 3
	YOLOv5			84.2	0.54	60	2.681e − 3
	SSD			87.2	0.59	73	3.45e − 3
	Proposed method			98.84	0.98	187.52	4.256e − 2

Note: Bold values show the metrics obtained for the proposed methods.

The proposed method provided the advantages of both traditional and other DL–based approaches. Using RPN and multiscale strategy to extract features helped Faster R-CNN to achieve a remarkable accuracy of 98.84% and an F1 score of 0.98. In addition, one of the main challenges of this research is to identify and count the productions at the same time, which makes the problem more complex, and the obtained results show that Faster R-CNN presents better and higher performance than other methods. Moreover, while the computational cost of training the proposed method is higher than other approaches, this drawback is mitigated by the offline nature of the training process. However, the testing phase boasts an execution time of under one second, making the model highly suitable for real-time applications. This comparison can be summarized as follows:

1. P–R curve: As shown in Figure 11, the proposed model can provide high P at all R levels, which reveals that it is more adapted to pharmaceutical products. It also has good R, suggesting that it can detect a significant number of objects.

2. Accuracy and F1 score: As illustrated in Table 2, Faster R-CNN achieves better accuracy and F1 score. This level of P is critical in pharmaceutical warehousing, where even small classification errors can lead to operational inefficiencies or safety risks.

3. Robust feature: Unlike YOLO and SSD, Faster R-CNN incorporates a RPN that generates high-quality region proposals. This ensures more accurate object localization and classification, which is especially valuable for identifying pharmaceutical products with diverse shapes, sizes, and orientations.

4. Efficient inference time: While Faster R-CNN has a higher computational cost during training, its testing time is under one second, which is comparable to YOLO and SSD. Since the training process is conducted offline, it does not impact real-time operations, making the method practical for deployment.

[figure(s) omitted; refer to PDF]

3.3. Implementation Results

To check the proposed algorithm’s performance in the pharmacy warehouse’s real environment, we designed software in a Python environment. To run the software, having the developed model file and the software file in a separate folder is enough. This software is designed so that the user can take a picture of the desired drugs on the shelf by pressing the button and view and record the name of the drug and its quantity in real time. Image acquisition was performed using an x11 webcam with a resolution of 3–5 megapixels, and the software was run on a computer with Intel® Core™ i5-4570 CPU @ 3.200 GHz with 8 GB of RAM. In this section, the results of the implementation of the proposed plan are presented. Figures 12 and 13 show the performance of the proposed developed model in a real pharmacy. The performance of the proposed system in a real environment can be seen in the Supporting video (available here).

[figure(s) omitted; refer to PDF]

4. Conclusion

Pharmacy warehousing is an undeniable operation in small and large pharmacies that imposes enormous costs on the shoulders of pharmacy owners every year. If warehouse management is implemented well, it can save significant resources for distribution/manufacturing companies by improving the complete working process and providing helpful business analysis. Today, AI has been proposed to reduce warehouse management costs, which can replace traditional warehouse management methods. This article introduces a method for identifying and counting the products available on pharmacy shelves using a deep neural network. For this purpose, a dataset including 6164 images with a bounding box has been prepared by using direct imaging and augmentation methods. Then, identifying and counting the items in the image were performed by using the pretrained Faster R-CNN network with ResNet50 backbone and choosing a region limit as the counting reference. The simulation results showed that the identification operation for the items in the images is performed with 98.84% accuracy and counting with 98% accuracy. The results of the project implementation showed that the proposed method effectively has the capability of intelligent warehouse management of a pharmacy by fixed and moving cameras.

Funding

The authors declare that this research received no specific funding. However, the publication cost will be covered by London South Bank University (LSBU).

References

[1] D. A. P. Pandian, "Artificial Intelligence Application in Smart Warehousing Environment for Automated Logistics," Journal of Artificial Intelligence and Capsule Networks, vol. 2019 no. 2, pp. 63-72, DOI: 10.36548/jaicn.2019.2.002, 2019.

[2] A. M. Atieh, H. Kaylani, Y. Al-Abdallat, "Performance Improvement of Inventory Management System Processes by an Automated Warehouse Management System," Procedia Cirp, vol. 41, pp. 568-572, DOI: 10.1016/j.procir.2015.12.122, 2016.

[3] J. Olivares-Aguila, W. ElMaraghy, H. ElMaraghy, "Human Capital Transformation for Successful Smart Manufacturing," Lecture Notes in Mechanical Engineering, pp. 871-878, DOI: 10.1007/978-3-030-90700-6_99, 2022.

[4] G. O. Akinola, O. T. Odesola, "Information and Communications Technology and Inventory Management Amongst Breweries in Nigeria," Journal of Information Systems Engineering and Business Intelligence, vol. 4 no. 1,DOI: 10.20473/jisebi.4.1.39-45, 2018.

[5] M. S. Et al, "Importance of an Efficient Warehouse Management System," Turkish Journal of Computer and Mathematics Education (TURCOMAT), vol. 12 no. 5, pp. 1185-1188, DOI: 10.17762/turcomat.v12i5.1784, 2021.

[6] S. M. Bragg, Inventory Best Practices, 2011.

[7] C. K. Lee, Y. Lv, K. K. H. Ng, W. Ho, K. L. Choy, "Design and Application of Internet of Things-Based Warehouse Management System for Smart Logistics," International Journal of Production Research, vol. 56 no. 8, pp. 2753-2768, DOI: 10.1080/00207543.2017.1394592, 2018.

[8] M. Torabizadeh, N. M. Yusof, A. Ma’aram, A. M. Shaharoun, "Identifying Sustainable Warehouse Management System Indicators and Proposing New Weighting Method," Journal of Cleaner Production, vol. 248,DOI: 10.1016/j.jclepro.2019.119190, 2020.

[9] E. Sisinni, A. Saifullah, S. Han, U. Jennehag, M. Gidlund, "Industrial Internet of Things: Challenges, Opportunities, and Directions," IEEE Transactions on Industrial Informatics, vol. 14 no. 11, pp. 4724-4734, DOI: 10.1109/tii.2018.2852491, 2018.

[10] G. Baryannis, S. Dani, G. Antoniou, "Predicting Supply Chain Risks Using Machine Learning: The Trade-Off between Performance and Interpretability," Future Generation Computer Systems, vol. 101, pp. 993-1004, DOI: 10.1016/j.future.2019.07.059, 2019.

[11] Í. R. da Costa Barros, T. P. Nascimento, "Robotic Mobile Fulfillment Systems: A Survey on Recent Developments and Research Opportunities," Robotics and Autonomous Systems, vol. 137,DOI: 10.1016/j.robot.2021.103729, 2021.

[12] Y. Zhang, J. Weng, Z. Ling, B. Pearson, X. Fu, "Bless: A Ble Application Security Scanning Framework," IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp. 636-645, .

[13] K. Baviskar, A. Bedse, S. Raut, N. Darapaneni, "Artificial Intelligence and Machine Learning‐Based Manufacturing and Drug Product Marketing," Bioinformatics Tools for Pharmaceutical Drug Product Development, pp. 197-231, DOI: 10.1002/9781119865728.ch10, 2023.

[14] J. Mao, H. Xing, X. Zhang, "Design of Intelligent Warehouse Management System," Wireless Personal Communications, vol. 102 no. 2, pp. 1355-1367, DOI: 10.1007/s11277-017-5199-7, 2018.

[15] B. Nitsche, F. Straube, M. Wirth, "Application Areas and Antecedents of Automation in Logistics and Supply Chain Management: a Conceptual Framework," Supply Chain Forum: International Journal, vol. 22 no. 3, pp. 223-239, DOI: 10.1080/16258312.2021.1934106, 2021.

[16] B. Downie, M. Gyöngyösi, C. Kuehl, "Artificial Intelligence in Materials Handling: How Machine Learning Tools Boost Warehouse Safety, Productivity and Cost-Effectiveness," Journal of Supply Chain Management, Logistics and Procurement, vol. 4 no. 1,DOI: 10.69554/qrxl2129, 2021.

[17] M. Du, J. Luo, S. Wang, S. Liu, "Genetic Algorithm Combined with BP Neural Network in Hospital Drug Inventory Management System," Neural Computing & Applications, vol. 32 no. 7, pp. 1981-1994, DOI: 10.1007/s00521-019-04379-3, 2020.

[18] S. Bandaru, T. Aslam, A. H. Ng, K. Deb, "Generalized Higher-Level Automated Innovization with Application to Inventory Management," European Journal of Operational Research, vol. 243 no. 2, pp. 480-496, DOI: 10.1016/j.ejor.2014.11.015, 2015.

[19] M. Qiu, Y. Song, "Predicting the Direction of Stock Market Index Movement Using an Optimized Artificial Neural Network Model," PLoS One, vol. 11 no. 5,DOI: 10.1371/journal.pone.0155133, 2016.

[20] A. Gunasekaran, E. W. Ngai, "Expert Systems and Artificial Intelligence in the 21st Century Logistics and Supply Chain Management," Expert Systems with Applications, vol. 41 no. 1,DOI: 10.1016/j.eswa.2013.09.006, 2014.

[21] T. Sustrova, "A Suitable Artificial Intelligence Model for Inventory Level Optimization," Trends Economics and Management, vol. 10 no. 25, pp. 48-55, DOI: 10.13164/trends.2016.25.48, 2016.

[22] D. Onoro-Rubio, R. J. López-Sastre, "Towards Perspective-free Object Counting with Deep Learning," Lecture Notes in Computer Science, vol. 14, pp. 615-629, DOI: 10.1007/978-3-319-46478-7_38, 2016.

[23] D. Zhang, L. G. Pee, L. Cui, "Artificial Intelligence in E-Commerce Fulfillment: A Case Study of Resource Orchestration at Alibaba’s Smart Warehouse," International Journal of Information Management, vol. 57,DOI: 10.1016/j.ijinfomgt.2020.102304, 2021.

[24] A. Krishna, L. Ravinchandra, R. Abdulla, K. F. Thang, "Smart Stock Management Control," International Journal of Applied Engineering Research, vol. 11 no. 1, pp. 492-500, 2016.

[25] S. Y. Yang, H. C. Jan, C. Y. Chen, M. S. Wang, "CNN-based QR Code Reading of Package for Unmanned Aerial Vehicle," Sensors, vol. 23 no. 10,DOI: 10.3390/s23104707, 2023.

[26] A. E. Wibowo, W. P. Lestari, "Pharmaceutical Storage Evaluation in Warehouse and Outlet of Apotek X Bantul Yogyakarta in 2018," 4th International Conference on Sustainable Innovation 2020–Health Science and Nursing (ICoSIHSN 2020), pp. 305-309, .

[27] D. K. Yimenu, A. M. Nigussie, T. Y. Workineh, "Assessment of Pharmaceutical Warehouse Management Practice: The Case of Private Pharmaceutical Wholesalers in Ethiopia," International Journal of Supply and Operations Management, vol. 8 no. 3, pp. 314-327, 2021.

[28] C. N. Guptha, M. G. Bhaskar, V. Meghasree, "Design of IoT Architecture for Order Picking in a Typical Warehouse," pp. 50-53, .

[29] W. T. Sung, C. Y. Lu, "Smart Warehouse Management Based on IoT Architecture," 2018 International Symposium on Computer, Consumer and Control (IS3C), pp. 169-172, 2018.

[30] M. G. Khan, N. U. Huda, U. K. U. Zaman, "Smart Warehouse Management System: Architecture, Real-Time Implementation and Prototype Design," Machines, vol. 10 no. 2,DOI: 10.3390/machines10020150, 2022.

[31] N. Macoir, J. Bauwens, B. Jooris, "Uwb Localization with Battery-Powered Wireless Backbone for Drone-Based Inventory Management," Sensors, vol. 19 no. 3,DOI: 10.3390/s19030467, 2019.

[32] F. Hamdi, A. Ghorbel, F. Masmoudi, L. Dupont, "Optimization of a Supply Portfolio in the Context of Supply Chain Risk Management: Literature Review," Journal of Intelligent Manufacturing, vol. 29 no. 4, pp. 763-788, DOI: 10.1007/s10845-015-1128-3, 2018.

[33] R. Hlioui, A. Gharbi, A. Hajji, "Replenishment, Production and Quality Control Strategies in Three-Stage Supply Chain," International Journal of Production Economics, vol. 166, pp. 90-102, DOI: 10.1016/j.ijpe.2015.04.015, 2015.

[34] R. Wang, W. Li, L. Zhang, "Blur Image Identification with Ensemble Convolution Neural Networks," Signal Processing, vol. 155, pp. 73-82, DOI: 10.1016/j.sigpro.2018.09.027, 2019.

[35] P. Tang, H. Wang, S. Kwong, "G-MS2F: GoogLeNet Based Multi-Stage Feature Fusion of Deep CNN for Scene Recognition," Neurocomputing, vol. 225, pp. 188-197, DOI: 10.1016/j.neucom.2016.11.023, 2017.

[36] R. Girshick, Fast R-CNN, 2015.

[37] S. Ren, K. He, R. Girshick, J. Sun, "Faster R-Cnn: Towards Real-Time Object Detection with Region Proposal Networks," Advances in Neural Information Processing Systems, vol. 28, 2015.

[38] H. Li, Y. Huang, Z. Zhang, "An Improved Faster R-CNN for Same Object Retrieval," IEEE Access, vol. 5, pp. 13665-13676, DOI: 10.1109/access.2017.2729943, 2017.

[39] L. Ding, H. Li, C. Hu, W. Zhang, S. Wang, "Alexnet Feature Extraction and Multi-Kernel Learning for Objectoriented Classification," The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 42, pp. 277-281, DOI: 10.5194/isprs-archives-xlii-3-277-2018, 2018.

[40] T. Les, M. Kruk, S. Osowski, "Automatic Recognition of Industrial Tools Using Artificial Intelligence Approach," Expert Systems with Applications, vol. 40 no. 12, pp. 4777-4784, DOI: 10.1016/j.eswa.2013.02.030, 2013.

[41] N. Huang, J. He, N. Zhu, X. Xuan, G. Liu, C. Chang, "Identification of the Source Camera of Images Based on Convolutional Neural Network," Digital Investigation, vol. 26, pp. 72-80, DOI: 10.1016/j.diin.2018.08.001, 2018.

[42] C. Siegfarth, T. Voegtle, C. Fabinski, "Comparison of Two Methods for 2d Pose Estimation of Industrial Workpieces in Images–CNN vs. Classical Image Processing System," The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 42, pp. 401-405, DOI: 10.5194/isprs-archives-xlii-1-401-2018, 2018.

[43] J. Yan, Z. Hao, R. Zhou, "A Quantitative Analysis Method Assisted by Image Features in Laser-Induced Breakdown Spectroscopy," Analytica Chimica Acta, vol. 1082, pp. 30-36, DOI: 10.1016/j.aca.2019.07.058, 2019.

[44] M. R. Hsieh, Y. L. Lin, W. H. Hsu, "Drone-based Object Counting by Spatially Regularized Regional Proposal Network," Proceedings of the IEEE International Conference on Computer Vision, pp. 4145-4153, .

[45] Z. Wu, X. Chen, Y. Gao, Y. Li, "Rapid Target Detection in High Resolution Remote Sensing Images Using YOLO Model," The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 42, pp. 1915-1920, DOI: 10.5194/isprs-archives-xlii-3-1915-2018, 2018.

[46] M. Mateen, J. Wen, S. Song, S. Song, Z. Huang, "Fundus Image Classification Using VGG-19 Architecture with PCA and SVD," Symmetry, vol. 11 no. 1,DOI: 10.3390/sym11010001, 2018.

[47] A. Selimović, B. Meden, P. Peer, A. Hladnik, "Analysis of Content-Aware Image Compression with VGG16," 2018 IEEE International Work Conference on Bioinspired Intelligence (IWOBI),DOI: 10.1109/iwobi.2018.8464188, .

[48] W. Liu, D. Anguelov, D. Erhan, "Ssd: Single Shot Multibox Detector," Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pp. 21-37, 2016.

[49] J. Jovel, R. Greiner, "An Introduction to Machine Learning Approaches for Biomedical Research," Frontiers of Medicine, vol. 8,DOI: 10.3389/fmed.2021.771607, 2021.

[50] A. Géron, Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2022.

[51] L. Tan, T. Huangfu, L. Wu, W. Chen, "Comparison of Yolo V3, Faster R-Cnn, and Ssd for Real-Time Pill Identification," 2021.

[52] Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, M. S. Lew, "Deep Learning for Visual Understanding: A Review," Neurocomputing, vol. 187, pp. 27-48, DOI: 10.1016/j.neucom.2015.09.116, 2016.

[53] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, "Deep Learning for Computer Vision: A Brief Review," Computational Intelligence and Neuroscience, vol. 2018,DOI: 10.1155/2018/7068349, 2018.

[54] R. Girshick, J. Donahue, T. Darrell, J. Malik, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580-587, DOI: 10.1109/cvpr.2014.81, .

[55] J. Vega, "Control System for Indoor Safety Measures Using a Faster R-CNN Architecture," Electronics, vol. 12 no. 11,DOI: 10.3390/electronics12112378, 2023.

[56] G. Rajeshkumar, M. Braveen, R. Venkatesh, "Smart Office Automation via Faster R-CNN Based Face Recognition and Internet of Things," Measurement: Sensors, vol. 27,DOI: 10.1016/j.measen.2023.100719, 2023.

[57] G. Gkioxari, R. Girshick, J. Malik, "Contextual Action Recognition with CNN," Proceedings of the IEEE International Conference on Computer Vision, pp. 1080-1088, .

[58] E. Hassan, M. Y. Shams, N. A. Hikal, S. Elmougy, "The Effect of Choosing Optimizer Algorithms to Improve Computer Vision Tasks: a Comparative Study," Multimedia Tools and Applications, vol. 82 no. 11, pp. 16591-16633, DOI: 10.1007/s11042-022-13820-0, 2023.

[59] H. E. Kim, A. Cosa-Linan, N. Santhanam, M. Jannesari, M. E. Maros, T. Ganslandt, "Transfer Learning for Medical Image Classification: a Literature Review," BMC Medical Imaging, vol. 22 no. 1,DOI: 10.1186/s12880-022-00793-7, 2022.

[60] P. M. Pereira, R. Fonseca-Pinto, R. P. Paiva, "Dermoscopic Skin Lesion Image Segmentation Based on Local Binary Pattern Clustering: Comparative Study," Biomedical Signal Processing and Control, vol. 59,DOI: 10.1016/j.bspc.2020.101924, 2020.

[61] J. Dou, C. Chen, P. Yang, "Genetic Scheduling and Reinforcement Learning in Multirobot Systems for Intelligent Warehouses," Mathematical Problems in Engineering, vol. 2015,DOI: 10.1155/2015/597956, 2015.