Risk Prediction Score for Thermal Mapping of Pharmaceutical Transport Routes in Brazil

Abstract

Background: The global pharmaceutical industry is crucial for providing medications but faces challenges in distributing products safely, especially in tropical and remote areas. Pharmaceuticals require careful transport control to maintain quality; therefore, manufacturers must adopt optimal distribution strategies to ensure product quality throughout the supply chain. The current research focused on creating a model to assess risk levels and predict risk categorization (low, moderate, and high) associated with thermal mapping across pharmaceutical transportation pathways. Methods: Data from a company for pharmaceutical logistics in Brazil were used. The data had 85,261 instances and six attributes (season, origin, destination, route, temperature, and temperature excursion). The dataset consisted of critical destinations, including the shipment time, cargo temperature, and route information. The classification algorithms (CART-Decision Tree, NB-Naive Bayes, and MP-Multilayer Perceptron) were used to build up a model of rules for predicting risk levels in thermal mapping routes; Results: The MP model presented the best performance, indicating a better application probability. The machine learning model is the basis for an automated risk prediction for routes of pharmaceutical transportation; Conclusions: the developed MP model might automatically predict risk during the distribution of pharmaceutical products, which might lead to optimizing time and costs.

1. Introduction

The global pharmaceutical industry is vital in ensuring access to essential medications for populations worldwide. However, numerous challenges have hampered the efficient and safe distribution of pharmaceutical products, particularly in regions characterized by tropical climates and distant rural areas [1]. Pharmaceutical products are susceptible to transport conditions and need strategic control to ensure quality during transport operations [2,3,4].

After the drug is manufactured, it is exposed to an environmental temperature outside the packaging range during a particular time. After this time of exposure, the product passes the long-term stability test stage to determine whether or not the excursion practiced initially impacts its stability until the end of its shelf life [5,6]. This requirement is a challenge for the manufacturer, as the warranty extends to end-use after passing through controls in the supply chain. The manufacturer must choose the best product distribution strategy to avoid degradation and maintain quality [3].

According to Klopott [7], 25% of the losses in the drug cold chain are attributed to issues encountered during transportation. The primary cause of loss or damage in pharmaceutical transport is the breakdown or malfunction of refrigerating equipment, which accounts for over half of all claims. The emergence of pharmaceutical cold chain logistics is based on refrigeration technology and the development of specialized logistics. The transport of products such as vaccines, injectables, tinctures, oral drugs, drugs for external use, and biological products can be described as pharmaceutical cold chain logistics. However, a low-temperature environment may not be sufficient for some drugs, which must be kept in cold conditions [8,9].

Thermal route mapping is a tool that gathers information and analyzes it by collecting data from the actual distribution routes [10]. Thermal mapping involves identifying the temperature range in which a pharmaceutical product must be transported and stored to maintain its efficacy and safety [11,12]. Companies operating in cold chain logistics protect consumers and improve security and reliability by performing real-time monitoring and implementing security management systems [13].

Previous studies have aimed to mitigate the departure from optimum temperature during the drug cold chain. Paul et al. (2020) [14] proposed the application of the Bayesian Belief Network (BBN) to effectively assess transportation disruption risks in supply chains, thereby assisting managers in predicting and formulating resilient strategies to address these risks. Zhou et al. [15] found that the spatial fuzzy multi-criteria evaluation approach efficiently assesses and maps maritime transportation risks, assisting authorities in developing practical plans to enhance navigation strategies in the international cold chain distribution.

With the development of information technology (IT), there have been attempts to incorporate digital-era solutions into pharmaceutical distribution systems. Faghih-Roohi et al. [16] proposed a group risk assessment approach for selecting pharmaceutical product shipping lanes using intuitionistic fuzzy numbers and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) built on Failure Modes and Effects Analysis (FMEA) to aggregate risk assessments from different experts and prioritize risks efficiently. At the same time, Shashi [17] developed a model for digitalizing pharmaceutical cold chain systems using IoT Digital Enabler. IoT-based digital enablers can improve pharmaceutical cold chain systems by addressing known and unknown constraints and enhancing temperature monitoring, transport, and storage. Moreover, Yang et al. [18] applied game theory to develop a reasonable revenue-sharing contract between medical institutions and logistics service providers to encourage decreased risk in cold chain transportation.

Another approach to minimize the risk of quality reduction is the combination of environmental sensor modules and wireless technology in transport vehicles for real-time data transmission, thus integrating the Internet of Things (IoT) and building intelligent systems for pharmaceutical logistics [19,20,21]. When a product leaves the supplier and enters the cold supply chain, checking and controlling its environment becomes challenging. Even though developed technologies, such as the Internet of Things (IoT), can effectively address this issue, IoT devices are vulnerable to data manipulation. To mitigate this risk, Bapatla et al. [22] employ IoT and blockchain technologies to effectively and steadily monitor and control the ambient parameters of cold chain shipments, thereby enhancing the reliability and safety of pharmaceuticals for consumers. However, the current literature does not provide an approach to forecast the risk in thermal mapping transport terrestrial routes.

Data mining is composed of the predictive modeling technique. Previous studies have proposed an intelligent supply chain management system for vaccine distribution using machine learning [20]. The technique extracts implicit database information, identifying and classifying new patterns [21,22,23]. The results obtained from data mining can be used in information management, information request processing, decision-making, and process control. The data contained in the databases are used to learn a particular target concept [23,24,25,26].

In predictive modeling and data analysis, three commonly employed algorithms are (1) Classification and Regression Trees (CART), (2) Naive Bayes (NB), and (3) Multilayer Perceptron (MP). These algorithms have distinct principles and procedures that make them suitable for various data types and predictive tasks. (1) CART is a decision tree algorithm for classification and regression tasks [27]. It works by recursively splitting the data into subsets based on the values of the input features, creating a tree-like structure of decisions. The algorithm selects the best split at each node based on a criterion for classification or mean squared error for regression. The tree is built until the data is sufficiently divided or a stopping criterion is met, such as a maximum tree depth or minimum node size. The final model is easy to interpret, as it can be visualized as a tree, with branches representing decision rules and leaves representing outcomes. (2) NB is a probabilistic algorithm based on Bayes’ theorem, which calculates the probability of a class given a set of features [28]. The “naive” assumption is that all features are conditionally independent given the class label, simplifying the probability computation. The algorithm estimates the probability distribution of the features within each class and then applies Bayes’ theorem to classify new data points based on these distributions. (3) MP is an artificial neural network consisting of multiple layers of neurons: an input layer, one or more hidden layers, and an output layer [29]. Each neuron in a layer is connected to neurons in the subsequent layer through weighted connections. The network learns to map inputs to outputs by adjusting these weights using backpropagation, which minimizes the error between the predicted and actual outputs. MP can capture complex non-linear relationships in data and is widely used in tasks such as image recognition, speech processing, and other areas requiring deep learning models. These three algorithms offer powerful predictive modeling tools, making them suitable for different data types and applications in scientific research.

According to Pezzola and Sweet [30], in the field of pharmaceutical regulation, most cross-national empirical studies have concentrated on intellectual property rights, often neglecting to examine the state’s capacity to regulate the pharmaceutical market and the differences in regulatory practices between countries, leading to difficulties in ensuring compliance during transport [1,2,30]. On the other hand, poor infrastructure, such as unreliable road networks and inadequate storage facilities, hampers the efficient transport of pharmaceuticals, especially temperature-sensitive products [31]. Brazil has uneven regional infrastructure development and relies heavily on trucks for freight [32]; therefore, particular focus should be applied to the road transportation of pharmaceuticals to ensure quality at the destination [33].

Automating risk analysis of temperature route specifications in the transport of pharmaceuticals allows temperature range at critical limits during the route to decision making. This automation can be done by applying machine learning training algorithms to classify the risk during thermal mapping routes. Therefore, the present study aimed to use the thermal mapping history of land freight transport routes to obtain a model for predicting the optimal temperature excursion. A risk assessment score was developed to predict different levels of risk in thermal mapping transport routes. Our study addresses a gap in pharmaceutical logistics by considering temperature excursion for packaging specifications on long drug distribution routes.

2. Materials and Methods

The present study was carried out with drugs with storage and transport specifications between 15 and 30 °C. Pharmacopeias and the thresholds for applicable ambient drug temperature ranges were used according to their temperature profile (Table 1).

2.1. Estimation of Optimal Temperature Occurrence as a Function of Thermal Mapping

A machine-learning model was developed to estimate the optimal temperature excursion. The study also compared the performance of all algorithms concerning their prediction abilities and model quality. The data was divided into the training and testing subsets in evaluating the models. For machine learning operations, 7078 data were split into 75% for training and 25% for exam sets. The flowchart (Figure 1) was used to identify the best training algorithm.

The pharmaceuticals were conditioned in thermal packs under suitable environmental conditions and transported in a dry cargo-box truck to Brazil’s northeast (within 100 km range of GPS coordinates Latitude −3°46′25.32″ S Longitude −38°34′29.28″ W) and south (within 100 km range of GPS coordinates 30°1′59.00″ S and 51°13′48.00″ W) regions by road. The parameters included in the tests to determine the optimal temperature excursion were that (1) for a temperature within 15 to 30 °C, there is an optimal temperature deviation (yes), and (2) for a temperature below <15 °C or >30 °C, there is not an optimal temperature deviation (no) [34].

2.2. Dataset Features and Data Mining Approach

The attributes utilized to construct the predictive model using the classification technique (modeling classification) comprise attributes on the route specification, including five attributes and 7078 instances, including the training set 5308 and the test set 1770 (Table 2 and Table 3).

The data set was divided into two parts (75% for training and 25% for testing) to avoid overlapping results or excessive adjustments when using the whole data set. The model was built using the training dataset (75%) and then validated using the testing dataset (25%). Data analysis included preprocessing steps such as normalization, noise removal, class balancing, and data transformation. To stratify the data set, the filter “stratified remove folds” was used to separate the data set between training and testing. The training and test data sets were loaded, stored, and transformed. After this preprocessing stage, a predictive model was inferred through training (75% of the dataset). The model was validated with the test set (25% of the data set) and subsequent application of NB and MP classification algorithms.

The following mining steps were performed from the dataset obtained in a commercial pharmaceutical logistics company: data selection, preprocessing, transformation, mining, analysis, and assimilation of results. The following attributes were discretized in the preprocessing stage: time/period (afternoon, night, dawn, and morning) and optimal temperature excursion (yes and no). Discretization reduces and simplifies data, making learning faster and providing more robust results [25]. Data preprocessing was performed in spreadsheets for further processing in the Weka data mining software version 3.8.3 [39,40,41,42,43]. After the preprocessing stage, a predictive model was inferred through training. The model was validated with the test set and subsequent application of NB and MP classification algorithms.

The NB and MP classification algorithms were applied to the training and test dataset to build a rule model for predicting the temperature excursion as a function of the route (northeast, south), period (dawn, morning, afternoon, night), and the season of the year (summer, winter). The technique employed was predictive modeling, a form of supervised learning where the class label for each training sample is provided. Test samples are randomly selected and are independent of training samples. Model validation was conducted using the cross-validation technique applied to the training set, ensuring that each record was used the same number of times for training and exactly once for testing.

2.3. Temperature Route Specification and Analysis

The criteria for the risk assessment in thermal route mapping considered the following: (1) whether the transported product entered a temperature variation and (2) the time it remained on the specified change on the transport route in the thermal mapping. When the temperature is between 15 and 30 °C, there is no temperature variation, and the risk is “low.” The risk is “moderate” when the temperature is lower than 15 or higher than 30 °C for more than or equal to 1 h. The risk is “high” when the temperature excursion exceeds 30 °C for over 1 h.

Data from a cold chain solutions company for pharmaceutical logistics was analyzed for two years by specifying temperature routes. The following data analysis steps were performed: data selection, preprocessing, transformation, mining, analysis, and interpretation of results. The preprocessing also included discretizing the attributes in classes that reduce and simplify the data, making the learning faster and the results denser [17,34]. The CART, NB, and MP classification algorithms were used to build a training and test dataset rules model. The performance of the algorithms in mitigating temperature excursions was evaluated using a range of assessment metrics: confusion matrix, sensitivity, accuracy, precision, the Matthews correlation coefficient (MCC), and the F value. The algorithm’s learning ability was assessed using the Kappa statistic, which measures the reliability of the classifications [44,45,46].

The following four terms are employed in the computation of metrics to gauge the model’s performance [46], and these terms are summarized in the confusion matrix: (1) True Positives (TP) are the positive values correctly labeled by the classifier; (2) True negatives (TN) refer to the negative tuples that the classifier has correctly identified; (3) False positives (FP) are the negative tuples wrongly labeled positive; and (4) False Negatives (FN) are the positive tuples erroneously labeled as negative. Equations (1) to (6) were used to evaluate the performance of the algorithms for accuracy, precision, sensitivity, Matthews Correlation Coefficient (MCC), and F value [43,46]:

(1)$False Positive Rate=1-{\displaystyle \frac{\left(TN\right)}{(FP+FN)}}$

(2)$Accuracy={\displaystyle \frac{(TP+TN)}{(TP+TN+FP+FN)}}$

(3)$Precision={\displaystyle \frac{\left(TP\right)}{(TP+FP)}}$

(4)$Sensitivity={\displaystyle \frac{\left(TP\right)}{(TP+FN)}}$

(5)$MCC={\displaystyle \frac{TP\times TN-FP\times FN}{\surd (TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$

(6)$F value=2\times ⌊{\displaystyle \frac{\left(Precision\times Sensitivity\right)}{\left(Precision+Sensitivity\right)}}⌋$

where TP = true positive; TN = true negative; FP = false positive; FN = false negative.

3. Results

3.1. Optimal Temperature Occurrence

The models designed to predict optimal temperature excursions during pharmaceutical transport demonstrated an enhanced overall performance for the MP model compared to Naive Bayes for training and testing (Table 4).

The MP model’s accuracy for the test subset was 93.7%, and the rates of incorrectly classified instances (5.2–6.3%, respectively) were lower and higher for the Kappa (85–81.9%, respectively). However, when applied to the test subset, the MP model slightly reduced performance. The two classification algorithms can be applied to estimate the optimal temperature excursion, as they presented performance above 90% (Figure 2).

The NB and MP models’ overall performance was superior to training, indicating greater learning in the test phase (Figure 2). These results indicate that the classes (yes and no) presented higher metrics results than the NB model (Table 5). The performance of the MP model also showed a lower rate of positive, false positive, and higher values in all the evaluated metrics (Table 6) compared to the NB model. The false-positive rate decreased by 3.7% in the “yes” class and 8.2% in the “no” class for the MP model.

The confusion matrix of the developed models (Table 7 and Table 8) showed that the NB model presented FP and FN numbers higher than those found for the MP model in training. In the test, there was an improvement in the results’ performance in the confusion matrix, showing fewer instances classified as FP and FN. There was a proportional increase in TP and TN. These values are related to the model’s precision, sensitivity, specificity, and quality.

3.2. Prediction and Risk Score

The performance of the CART, NB, and MP models were similar when analyzing accuracy, misclassified instances, and Kappa, with values between 91.1 and 99.3%, 0.7 and 0.9%, 95.3 and 96.2%, respectively (Table 9). The MP model presented better performance results by class when compared to the CART and NB models.

The higher hits can explain this result for the intermediate class “moderate risk” in precision (50%), sensitivity (12.3%), F value (19.7%), MCC (24.6%), and higher true-positive rate (12.3%) when compared to the results of the CART and NB models (Table 10).

The hits for the “low risk” and “high risk” classes were generally more efficient and better, which can be justified by the Kappa. However, the “moderate risk” class still needs adjustments and training with higher instances. The MP model (Table 11) presented fewer wrongly classified instances for “moderate risk,” classified 106 as “high risk” compared to 121 instances for CART and 122 for NB. It presented the highest number of hits of instances correctly classified as “moderate risk” for the MP. All models had similar success rates in the “low risk” and “high risk” classes.

4. Final Remarks

We propose an assessment score to predict risk in the thermal mapping of pharmaceutical transport routes in Brazilian conditions. Similar to the present study, previous research indicates that machine learning models may reduce logistics operation costs [2,47,48]. The cold chain literature typically pertains to transporting perishable products using thermal and refrigerated packaging methods, alongside logistics planning, to ensure the integrity of shipments is maintained [4,49,50,51]. Perishable products maintain chemical reactions attenuated due to low temperatures; however, delays and problems in transportation can have negative consequences [52,53].

Temperature route specification protocols are used for the thermal mapping of routes to ensure quality throughout the supply chain and predict risks during the transportation process [54]. However, it is challenging in developing countries due to poor road infrastructure, mainly in rural areas. The temperature data outside the acceptance range needs electronic monitoring [55]. This study might contribute to the automation of this risk prediction through machine learning to predict models and evaluate their performance for application in new data. Three models were tested to predict risk during thermal mapping, including CART, NB, and MP. The MP model was superior to the CART and NB algorithms because it performed better in the sensitivity metric or true positive rate, especially for intermediate classes in the risk classification task in route specification. In practice, this means that predicting risk with greater sensitivity at an intermediate level would help to avoid high-risk thermal mapping routes.

The infrastructure required to transport pharmaceuticals is a huge challenge. Violations of the cold chain may affect quality, making therapeutics harmful or ineffective. Predicting the risk of departure from optimal medication transport in developing countries requires careful consideration of each country’s unique challenges [56].

As a logistics solution, the MP model outperformed the three models tested. The hit rates by class, mainly by higher hits in the intermediate class (moderate risk) prediction, justify a better application probability. This solution may contribute to the automatic prediction of risk during transport in thermal mapping and, consequently, optimize time and costs in the distribution of pharmaceutical products [4,49,51,52]. Solutions based on algorithms can provide opportunities to optimize the pharmaceutical supply chain’s complex processes.

The assessment score to predict risk in the thermal mapping of pharmaceutical transport routes is essential for risk management in specifying temperature routes and pharmaceutical logistics; consequently, it contributes to improving the chain. The MP model has great application potential and presents more accurate results in modeling. It ensures learning about risk management while transporting pharmaceutical products. The strategic and managerial bias, based on data analysis in machine learning, guides decision-making and manages risks during the transport routes of pharmaceutical products.

Predictive modeling leverages historical data to forecast future outcomes, assess whether the temperature of pharmaceuticals during transportation remained within specified limits, and identify and predict varying levels of risk in heat mapping along transport routes. These models are applicable in classifying the routes of various cold chain products based on tested specificity, enhancing predictive risk management analysis. These models can support more informed decision-making by considering potential future scenarios, reducing risk, and improving operational efficiency.

Author Contributions

The concept and design, C.G.M. and N.D.d.S.L. Acquisition of data, Mangini, C.G.M. Analysis and interpretation of data, C.G.M., I.d.A.N. and N.D.d.S.L. Drafting of the manuscript, C.G.M. and N.D.d.S.L. Critical paper revision for important intellectual content, C.G.M., I.d.A.N. and N.D.d.S.L. Supervision, I.d.A.N. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

Data is available in the Supplementary Files.

Acknowledgments

CAPES-Brazilian Agency for Support and Evaluation of Graduate Education for the first author’s doctoral scholarship.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Machine learning training and testing process flowchart (Adapted from [38]).

Figure 2. Comparison of the overall performance of the Naive Bayes and Multilayer Perceptron models.

Table 1

Pharmacopeias and determine applicable ambient drug temperature ranges according to the temperature profile.

Reference	Norms
[34]	Normal storage conditions. Storage in dry, well-ventilated places at temperatures of 15–25 °C or, depending on climate conditions, until reaches 30 °C. Do not store above 30 °C (from +2 to +30 °C). Do not store above 25 °C (from +2 to +25 °C).
[35]	Ambient temperature: The temperature that prevails in a working environment. Controlled ambient temperature: The temperature is maintained with the aid of thermostats in an ambient of 20–25 °C.
[36]	Storage at a continuous temperature of 25 °C during real-time stability studies, i.e., covers the temperature likely to be encountered in ambient conditions throughout Europe, including real-time tours from 25 °C.
[37]	Standard temperature, normal temperature, room temperature, and warm are defined as 20 °C, 15–25 °C, 1–30 °C, and 30–40 °C, respectively.
[38]	Ambient temperature; 15 °C to 25 °C.

Table 2

Criteria for temperature excursion test.

Temperature (°C)	Optimal Temperature Deviation
15 to 30	yes
<5 to >30	no

Source: [34].

Table 3

Attributes that make up the model training dataset.

Training Set (75% of the Total Dataset)
Attributes	Label	Count	Type	Total Instances *
Seasons	winter	2590	nominal	5308
	summer	2718
Period	evening	1339	nominal	5308
	night	1349
	dawn	1342
	morning	1278
Destination region	Northeast Region	2590	nominal	5308
	South Region	2718
Temperature	minimum	6.30	numeric	5308
	maximum	41.80
	mean	20.71
	StdDev	6.09
Optimal temperature excursion(Target)	yes	4023	nominal	5308
	no	1285

* The training set is 75% of the total dataset. StdDev = standard deviation.

Table 4

Attributes that make up the dataset in the test of models.

Test Set (25% of the Total Dataset)
Attributes	Label	Count	Type	Total Instances *
Seasons	winter	2590	nominal	5308
	summer	2718
Period	evening	440	nominal	1770
	night	450
	dawn	452
	morning	428
Destination region	Northeast Region	866	nominal	1770
	South Region	904
Temperature	minimum	6.30	numeric	1770
	maximum	41.30
	mean	20.65
	StdDev	6.11
Optimal temperature excursion(Target)	yes	1342	nominal	1770
	no	428

* The training set is 25% of the total dataset. StdDev = standard deviation.

Table 5

Performance of temperature excursion classification models in the training subset.

	Naive Bayes Model			Multilayer Perceptron Model
Training	Detailed Accuracy by Class			Detailed Accuracy by Class
Metrics	Yes (%)	No (%)	WA (%)	Yes (%)	No (%)	WA (%)
True Positive Rate	90.5	74.7	86.7	98.8	82.1	94.8
False Positive Rate	25.3	09.5	21.5	17.9	1.2	13.9
Precision	91.8	71.5	86.9	94.5	95.6	94.8
Recall	90.5	74.7	86.7	98.8	82.1	94.8
F-Measure	91.1	73.1	86.8	96.6	88.4	94.6
MCC	64.2	64.2	64.2	85.4	85.4	85.4

MCC = Matthews correlation coefficient; WA = weighted average.

Table 6

Performance of temperature excursion classification models in the test subset.

	Naive Bayes Model			Multilayer Perceptron Model
Training	Detailed Accuracy by Class			Detailed Accuracy by Class
Metrics	Yes (%)	No (%)	WA (%)	Yes (%)	No (%)	WA (%)
True Positive Rate	90.5	74.7	86.7	98.8	82.1	94.8
False Positive Rate	25.3	90.2	75.5	86.6	98.4	79.2
Precision	91.8	24.5	9.8	21.0	20.8	1.6
Recall	90.5	92.0	71.0	86.9	93.7	93.9
F-Measure	91.1	90.2	75.5	86.6	98.4	79.2
MCC	64.2	91.1	73.2	86.7	96.0	85.9

MCC = Matthews correlation coefficient; WA = weighted average.

Table 7

Naive Bayes model classification confusion matrix for training.

Naive Bayes Model—Training
Yes (n)	No (n)	Total (n)	Classified as
3640 (TP)	383 (FP)	4023	yes
325 (FN)	960 (TN)	1285	no
3965	1343	5308
Multilayer Perceptron Model—Training
Yes (n)	No (n)	Total (n)	Classified as
3975 (TP)	48 (FP)	4023	yes
230 (FN)	1055 (TN)	1285	no
4205	1103	5308

Table 8

Multilayer Perceptron model classification confusion matrix for testing.

Naive Bayes Model—Training
Yes (n)	No (n)	Total (n)	Classified as
1210 (TP)	132 (FP)	1342	yes
105 (FN)	323 (TN)	428	no
1315	455	1770
Multilayer Perceptron Model—Training
Yes (n)	No (n)	Total (n)	Classified as
1320 (TP)	22 (FP)	1342	yes
89 (FN)	339 (TN)	428	no
1409	361	1770

Table 9

The general performance of prediction models for risk assessment in thermal mapping.

Classifier Model	CART Decision Tree	Naive Bayes	Multilayer Perceptron
Accuracy (%)	99.3	99.1	99.3
Instances classified incorrectly (%)	0.7	0.9	0.7
Kappa Statistics (%)	96.2	95.3	96.2

Table 10

Performance of detailed prediction models by class of risk levels in temperature pathways.

Model	CART Decision Tree
Performance Metrics	Low Risk	Moderate Risk	High Risk
True Positive Rate (%)	99.8	0.8	99.8
False Positive Rate (%)	0.0	0.0	0.80
Precision (%)	100.0	20.0	93.4
Sensitivity (%)	99.8	0.8	99.8
F value (%)	99.9	1.6	96.5
MCC (%)	99.3	3.9	96.1
Model	Naive Bayes
Performance Metrics	Low Risk	Moderate Risk	High Risk
True Positive Rate (%)	99.6	0	100.0
False Positive Rate (%)	0	0	1
Precision (%)	100.0	0	91.8
Sensitivity (%)	99.6	0	100.0
F value (%)	99.8	0	95.7
MCC (%)	98.4	0.1	95.3
Model	Multilayer Perceptron
Performance Metrics	Low Risk	Moderate Risk	High Risk
True Positive Rate (%)	99.9	12.3	99.1
False Positive Rate (%)	0.3	0.1	0.7
Precision (%)	100.0	50.0	94.0
Sensitivity (%)	99.9	12.3	99.1
F value (%)	99.9	19.7	96.5
MCC (%)	99.2	24.6	96.1

MCC: Matthews correlation coefficient.

Table 11

Confusion matrix of models for predicting risk levels in temperature routes.

CART Decision Tree
Low Risk (n)	Moderate Risk (n)	High Risk (n)	Total (n)	Classified as
19,038	0	29	19,067	Low risk
0	1	121	122	Moderate risk
1	4	2122	2127	High risk
19,039	5	2272	21,316
Naive Bayes
Low Risk (n)	Moderate Risk (n)	High Risk (n)	Total (n)	Classified as
19,000	0	67	19,067	Low risk
0	0	122	122	Moderate risk
0	1	2126	2127	High risk
19,000	1	2315	21,316
Multilayer Perceptron
Low Risk (n)	Moderate Risk (n)	High Risk (n)	Total (n)	Classified as
19,039	0	28	19,067	Low risk
1	15	106	122	Moderate risk
5	15	2107	2127	High risk
19,045	30	2241	21,316

Supplementary Materials

The Supplementary file can be downloaded at: https://www.mdpi.com/article/10.3390/logistics8030084/s1.

References

1. Levin, S.R. Improving distribution of pharmaceuticals in developing countries: A case study of The Gambia project. J. Technol. Transfer; 1987; 11, pp. 1-8. [DOI: https://dx.doi.org/10.1007/BF02174374]

2. Bogataj, M.; Bogataj, L.; Vodopivec, R. Stability of perishable goods in cold logistic chains. Int. J. Prod. Econ.; 2005; 93, pp. 345-356. [DOI: https://dx.doi.org/10.1016/j.ijpe.2004.06.032]

3. Ammann, C. Stability studies need to define the handling and transport conditions of sensitive pharmaceutical or biotechnological products. AAPS PharmSciTech; 2011; 12, pp. 1264-1275. [DOI: https://dx.doi.org/10.1208/s12249-011-9684-0] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/21948319]

4. Rodrigue, J.P.; Notteboom, T. The cold chain and its logistics. The Geography of Transport Systems; Routledge: New York, NY, USA, 2020; Available online: https://transportgeography.org/contents/applications/cold-chain-logistics/ (accessed on 20 February 2024).

5. Kim, K.; Kim, H.; Kim, S.K.; Jung, J.Y. i-RM: An intelligent risk management framework for context-aware ubiquitous cold chain logistics. Expert Syst. Appl.; 2016; 46, pp. 463-473. [DOI: https://dx.doi.org/10.1016/j.eswa.2015.11.005]

6. Wilson, E.; Zhu, C.; Galea, S.; Marko, A.; Urdaneta, V.V.; Straus, W. Turning up the heat: Effect of new vaccine for children’s (VFC) program recommendations for use of temperature monitors upon incorrect product storage adverse event reporting. Vaccine; 2018; 36, pp. 1516-1520. [DOI: https://dx.doi.org/10.1016/j.vaccine.2017.10.059]

7. Klopott, M. Loss and damage analysis in international transport of pharmaceutical products. Eur. Res. Stud.; 2021; 24, pp. 799-807. [DOI: https://dx.doi.org/10.35808/ersj/2772]

8. Chen, K.Y.; Shaw, Y.C. Applying back propagation network to cold chain temperature monitoring. Adv. Eng. Inform.; 2011; 25, pp. 11-22. [DOI: https://dx.doi.org/10.1016/j.aei.2010.05.003]

9. Sharma, S.; Pai, S. Analysis of operating effectiveness of a cold chain model using Bayesian networks. Bus. Process Manag. J.; 2015; 21, pp. 722-742. [DOI: https://dx.doi.org/10.1108/BPMJ-10-2014-0105]

10. Kaur, N.; Gupta, D.; Singla, R.; Bharadwaj, A.; Kavit, A.; Mansoor, W. Thermal Aware Routing Protocols in, W.B.A.N. Available online: https://doi.org/10.1109/ICSPIS53734.2021.9652442 (accessed on 10 November 2023).

11. Lippi, G.; Lima-Oliveira, G.; Nazer, S.C.; Moreira, M.L.L.; Souza, R.F.M.; Salvagno, G.L.; Montagnana, M.; Scartezini, M.; Picheth, G.; Guidi, G.C. Suitability of a transport box for blood sample shipment over a long period. Clin. Biochem.; 2011; 44, pp. 1028-1029. [DOI: https://dx.doi.org/10.1016/j.clinbiochem.2011.05.028]

12. Kulkarni, N.S.; Niranjan, S. Multi-Echelon Network Optimization of Pharmaceutical Cold Chains: A Simulation Study. Available online: https://www.researchgate.net/profile/Niranjan-Kulkarni-2/publication/266278288_Multi-echelon_Network_Optimization_of_Pharmaceutical_Cold_Chains/links/55c200c808aeb5e0c584d759/Multi-echelon-Network-Optimization-of-Pharmaceutical-Cold-Chains.pdf (accessed on 30 November 2023).

13. Ou, C.M.; Tu, J.F. The WSN and 3G/NFC embedded into IoV (Internet- of-Vehicle) fulfill cold chain logistics. Microsyst. Technol.; 2018; 24, pp. 3977-3983. [DOI: https://dx.doi.org/10.1007/s00542-017-3594-3]

14. Paul, S.; Kabir, G.; Ali, S.; Zhang, G. Examining transportation disruption risk in supply chains: A case study from Bangladeshi pharmaceutical industry. RTBM; 2020; 37, 100485. [DOI: https://dx.doi.org/10.1016/j.rtbm.2020.100485]

15. Zhou, X.; Cheng, L.; Li, M. Assessing and mapping maritime transportation risk based on spatial fuzzy multi-criteria decision making: A case study in the South China sea. Ocean Eng.; 2020; 208, 107403. [DOI: https://dx.doi.org/10.1016/j.oceaneng.2020.107403]

16. Faghih-Roohi, S.; Akçay, A.; Zhang, Y.; Shekarian, E.; Jong, E. A group risk assessment approach for the selection of pharmaceutical product shipping lanes. Int. J. Prod. Econ.; 2020; 229, 107774. [DOI: https://dx.doi.org/10.1016/j.ijpe.2020.107774]

17. Shashi, D. Digitalization of pharmaceutical cold chain systems using IoT Digital enabler. IJEAT; 2022; 11, pp. 133-137. [DOI: https://dx.doi.org/10.35940/ijeat.E3622.0611522]

18. Yang, Z.; Zhao, K.; Guo, Q. Mathematical Problems in Engineering Pharmaceutical Cold Chain Transportation Decision Making under the Government’s Reward-Penalty Mechanism: A Perspective of Evolutionary Game Theory. Math. Probl. Eng.; 2022; 2022, 1555042. [DOI: https://dx.doi.org/10.1155/2022/1555042]

19. Li, C.M.; Nien, C.C.; Liao, J.L.; Tseng, Y.C. Development of wireless sensor module and network for temperature monitoring in cold chain logistics. Proceedings of the IEEE International Conference on Wireless Information Technology and Systems (ICWITS); Maui, HI, USA, 11–16 November 2012; pp. 1-4. [DOI: https://dx.doi.org/10.1109/ICWITS.2012.6417745]

20. Öcal, M.; Kaya, İ.A. Food Safety and GIS Applications. Available online: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7248098&casa_token=fo29ue4PzWQAAAAA:_c2SukQMiP91Bu45t9tyP1XOSjFbO-cAl8Uk4D1KhDiyOVH08rd_kBe078YVuOY4vMlue0CP_vGqWw&tag=1 (accessed on 24 June 2023).

21. Hu, H.; Xu, J.; Liu, M.; Lim, M.K. Vaccine supply chain management: An intelligent system utilizing blockchain, IoT and machine learning. J. Bus. Res.; 2023; 156, 113480. [DOI: https://dx.doi.org/10.1016/j.jbusres.2022.113480] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36506475]

22. Bapatla, A.; Mohanty, S.; Kougianos, E.; Puthal, D. PharmaChain 2.0: A Blockchain framework for secure remote monitoring of drug environmental parameters in pharmaceutical cold supply chain. Proceedings of the 2022 IEEE International Symposium on Smart Electronic Systems (iSES); Warangal, India, 18–22 December 2022; pp. 185-190. [DOI: https://dx.doi.org/10.1109/iSES54909.2022.00046]

23. Frank, E.; Hall, M.; Holmes, G.; Kirkby, R.; Pfahringer, B.; Witten, I.H.; Trigg, L. Weka: A machine learning workbench for data mining. Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers; Maimon, O.; Rokach, L. Springer: Berlin, Germany, 2012.

24. Ngai, E.W.T.; Xiu, L.; Chau, D.C.K. Application of data mining techniques in customer relationship management: A literature review and classification. Expert Syst. Appl.; 2009; 36, pp. 2592-2602. [DOI: https://dx.doi.org/10.1016/j.eswa.2008.02.021]

25. Larose, D.T.; Larose, C.D. Data Mining and Predictive Analytics; John Wiley & Sons: Hoboken, NJ, USA, 2015.

26. Naik, A.; Samant, L. Correlation review of classification algorithm using Data Mining Tool: Weka, Rapidminer, Tanagra, Orange, and Knime. Procedia Comput. Sci.; 2016; 85, pp. 662-668. [DOI: https://dx.doi.org/10.1016/j.procs.2016.05.251]

27. Bar-Hen, A.; Gey, S.; Poggi, J.M. Influence measures for CART classification trees. J. Classif.; 2015; 32, pp. 21-45. [DOI: https://dx.doi.org/10.1007/s00357-015-9172-4]

28. Yang, Z.; Ren, J.; Zhang, Z.; Sun, Y.; Zhang, C.; Wang, M.; Wang, L. New three-way incremental Naive Bayes classifier. Electronics; 2023; 12, 1730. [DOI: https://dx.doi.org/10.3390/electronics12071730]

29. Almeida, L.B. Multilayer perceptrons. Handbook of Neural Computation; CRC Press: Boca Raton, FL, USA, 2020; pp. C1-C2.

30. Pezzola, A.; Sweet, C. Global pharmaceutical regulation: The challenge of integration for developing states. Glob. Health; 2016; 12, 85. [DOI: https://dx.doi.org/10.1186/s12992-016-0208-2]

31. Njuguna, M.; Mairura, C.; Ombui, K. Influence of cold chain supply logistics on the safety of vaccines. A Case of Pharmaceutical Distributors in Nairobi County. Dev. Ctry. Stud.; 2015; 5, pp. 91-108.

32. De Souza, F.L.U.; Larranaga, A.M.; Palma, D.; Pitombo, C.S. Modeling travel mode choice and characterizing freight transport in a Brazilian context. Transp. Lett.; 2022; 14, pp. 983-996. [DOI: https://dx.doi.org/10.1080/19427867.2021.1976011]

33. da Silva, T.D.; dos Santos Macêdo, D.C.; dos Santos Lucena, J.N.; da Silva, R.M.F.; de Lira Soares, L.A.; Pereira, M.C.; Paulino, S.J.S.; da Silva, B.P.; Lucena, M.L.C. Systematic assessment of the drugs cold chain in Brazil by pharmaceutical professionals. Res. Soc. Dev.; 2021; 10, e5010716144. [DOI: https://dx.doi.org/10.33448/rsd-v10i7.16144]

34. WHO. World Health Organization Technical Report Series n◦ 908-2003-Annex 9. Guide to Good Storage Practices for Pharmaceuticals—Appendix. Storage and Labeling Conditions. Available online: https://iris.who.int/bitstream/handle/10665/42613/WHO_TRS_908.pdf?sequence=1 (accessed on 20 November 2023).

35. USPNF Packaging and Storage Requirements. Available online: https://www.uspnf.com/sites/default/files/usp_pdf/EN/USPNF/revisions/659_rb_notice.pdf (accessed on 10 November 2023).

36. EMA Guideline CPMP/QWP/609/96/Rev2-Declaration of Storage Conditions in the Product Information of Medicinal Products-2007. Available online: https://www.ema.europa.eu/en/declaration-storage-conditions-medicinal-products-particulars-and-active-substances-annex-scientific-guideline (accessed on 10 January 2024).

37. Japanese Pharmacopeia. Edition XVII-General Notice Page 2-16. Available online: https://www.mhlw.go.jp/content/11120000/000945683.pdf (accessed on 22 April 2024).

38. British Pharmacopeia Edition. General Notices. Part III under Temperature-2020. Available online: https://www.pharmacopoeia.com/compliance.org/guidemgr/files/060996en.pdf (accessed on 22 April 2024).

39. Uçar, M.K.; Nour, M.; Sindi, H.; Polat, K. The effect of training and testing process on machine learning in biomedical datasets. Math. Prob Eng.; 2020; 2020, 2836236. [DOI: https://dx.doi.org/10.1155/2020/2836236]

40. Holmes, G.; Donkin, A.; Witten, I.H. Weka: A Machine Learning Workbench. Available online: https://researchcommons.waikato.ac.nz/bitstream/handle/10289/1138/uow-cs-wp-1994-09.pdf?sequence=1 (accessed on 10 December 2023).

41. Witten, I.H.; Frank, E.; Trigg, L.E.; Hall, M.A.; Holmes, G.; Cunningham, S.J. Weka: Practical Machine Learning Tools and Techniques with Java Implementations. Available online: https://researchcommons.waikato.ac.nz/items/30f20075-25f4-440b-ad4a-4c95803d4391 (accessed on 12 November 2023).

42. Hall, M.; Frank, E.; Holmes, G.; Pfahringer, B.; Reutemann, P.; Witten, I.H. The WEKA data mining software: An update. ACM SIGKDD Explor. Newsl.; 2009; 11, pp. 10-18. [DOI: https://dx.doi.org/10.1145/1656274.1656278]

43. Markov, Z.; Russell, I. An introduction to the WEKA data mining system. ACM SIGCSE Bull.; 2006; 38, pp. 367-368. [DOI: https://dx.doi.org/10.1145/1140123.1140127]

44. Carletta, J. Assessing agreement on classification tasks: The Kappa statistic. Compu. Linguist.; 1996; 22, pp. 249-254.

45. Bowes, D.; Hall, T.; Gray, D. Comparing the Performance of Fault Prediction Models Which Report Multiple Performance Measures: Recomputing the Confusion Matrix. Available online: https://dl.acm.org/doi/abs/10.1145/2365324.2365338?casa_token=e4w3hcKrPtQAAAAA%3ACw9vh63aQAXZY2Cit9js7Inu37MpzSRiHG_eB9UFn6xWMu0OMzIVr9xkOdS7Av5ZmjhHYRIF3hIC9A (accessed on 23 March 2024).

46. Han, J.; Kamber, M.; Pei, J. Classification: Basic Concepts. The Morgan Kaufmann Series in Data Management Systems; Han, J.; Kamber, M.; Pei, J. Elsevier Science Ltd.: Geneva, The Netherlands, 2012; pp. 327-391.

47. Gessner, G.H.; Volonino, L.; Fish, L.A. One-up, one-back ERM in the food supply chain. Inf. Syst. Manag.; 2007; 24, pp. 213-222. [DOI: https://dx.doi.org/10.1080/10580530701404561]

48. Lee, H.; Jo, S.K.; Lee, N.; Lee, H.W. A method for co-existing heterogeneous IoT environments based on compressive sensing. Proceedings of the 2016 18th International Conference on Advanced Communication Technology (ICACT); PyeongChang, Republic of Korea, 31 January–3 February 2016; pp. 206-209. [DOI: https://dx.doi.org/10.1109/ICACT.2016.7423330]

49. Raab, V.; Petersen, B.; Kreyenschmidt, J. Temperature monitoring in meat supply chains. Br. Food J.; 2011; 113, pp. 1267-1289. [DOI: https://dx.doi.org/10.1108/00070701111177683]

50. Uthayakumkar, R.; Priyan, S. Pharmaceutical supply chain and inventory management strategies: Optimization for a pharmaceutical company and a hospital. Oper. Res. Health Care; 2013; 2, pp. 52-64. [DOI: https://dx.doi.org/10.1016/j.orhc.2013.08.001]

51. Campos, Y.; Villa, J.L. Technologies Applied in the Monitoring and Control of the Temperature in the Cold Chain. Available online: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8588118&casa_token=Ldm0eQnWKRsAAAAA:mU4cza6QxjsvQhe08iCpPDl3-LmtgGu8HJOmgrmzE2g4fqO6Zt3ZFS64lvv8Vdq22o0_JVrdX0ZYHA (accessed on 6 October 2023).

52. Montanari, R. Cold chain tracking: A managerial perspective. Trends Food Sci. Technol.; 2008; 19, pp. 425-431. [DOI: https://dx.doi.org/10.1016/j.tifs.2008.03.009]

53. Kumar, N.; Jha, A. Temperature excursion management: A novel approach of quality system in pharmaceutical industry. Saudi Pharm. J.; 2017; 25, pp. 176-183. [DOI: https://dx.doi.org/10.1016/j.jsps.2016.07.001] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28344467]

54. Bhatnagar, A.; Gupta, V.; Tandon, P.; Saksena, T.; Ranjan, A.; Gandhi, P.; Garcha, S.C.; Kapoor, A. Last Mile Delivery of Cold Chain Medicines—Challenges and Recommendations. IJPBR; 2018; 6, pp. 34-41. [DOI: https://dx.doi.org/10.30750/ijpbr.6.1.6]

55. Brazil. Collegiate Board Resolution—RDC no. 430 of October 2020. Available online: https://www.in.gov.br/en/web/dou/-/resolucao-de-diretoria-colegiada-rdc-n-430-de-8-de-outubro-de-2020-282070593 (accessed on 10 September 2023).

56. De Boeck, K.; Decouttere, C.; Vandaele, N. Vaccine distribution chains in low- and middle-income countries: A literature review. Omega; 2020; 97, 102097. [DOI: https://dx.doi.org/10.1016/j.omega.2019.08.004]