Performance Evaluation of Reverse Logistics: Opportunities for Future Research

Abstract

This study aimed to analyze the characteristics of the scientific research that approaches Reverse Logistics (RL) from the perspective of Performance Evaluation (PE). For this purpose, ProKnow-C was used to select 21 articles to compose a Bibliographical Portfolio (BP). Among the results, we mention Govindan, Sarkis, Zhu and Lai as prominent authors. By analyzing the articles, it was perceived that most of them perform the RL and PE fields separately and present tools that consider the alignment of the indicators with the strategy. Also, the articles lack of a structured PE process that could serve as a subsidy to the practice of logistics management. The results present that there is a theoretical gap in the literature of a PE model of RL.

1. Introduction

Reverse logistics is an increasingly discussed subject both under supply chain management literature and the environmental theme [1]. The reverse logistics process consists of a series of activities required to collect an obsolete, unused, or disposal product, in order to generate revenue through recycling or resale in the secondary market, but also to guarantee environmentally correct disposal [2]. In the last decade, environmental awareness has become a liability for many companies in the supply chain and is mostly fostered by government regulations and under client perspectives on social and environmental issues [3,4].

The dissemination of knowledge of the practices of reverse logistics management has made companies aware not only of the impact they are causing but also the loss of new business opportunities [5]. With the generation of RL (Reverse Logistics) knowledge, combined with the high level of information and indicators on which decision-making is based, the Reverse Logistics Performance Assessment can be seen as a crucial factor in guiding strategic, tactical and operational decisions.

Developing metrics for measuring the performance of a supply chain is a difficult problem and it becomes more complex when taking the reverse logistics chain in account, given the difficulties in operating and coordinating the flow of materials and information [6]. Among the questions that need to be addressed for the evaluation of performance: What has to be measured? How often does it have to be measured? How to integrate several individual measurements into a measurement system [7]? Neely [8] has reported the difficulty of establishing performance metrics due to the singularities of each organization and that, even when using unique metrics adequate for a given scenario, they should be periodically reviewed and, when necessary, new ones should be incorporated into measuring systems.

Effective performance management is an important aspect of the reverse logistics initiative in corporations [9]. Organizations are dealing with many uncertainties when facing their market, characterized with a high rate of return due to the expansion of portfolios and shorter life cycles of products, which generates a decisive impact on the strategies of companies [6]. There are two main issues in this context: (i) Evaluation of various strategies, processes and resources to achieve the objectives and develop measures; and (ii) how the organization should prioritize the initiatives and returns of stakeholders that directly impact them.

In addition to the need for awareness and sustainable practices, reverse logistics operators still suffer from a lack of knowledge of the process involved, which leads to various problems such as loss-allocation processes, little information exchange between different providers (members of sector), lack of knowledge diffusion, and a small participation of the academic community in the development of the sector [10].

Based on these arguments, it is evident the need for knowledge of PE of RL in organizations, providing support and robustness for decision making. Therefore, the question that guides this study: What are the opportunities of research on the subject of reverse logistics management from the perspective of performance?

As a way of addressing this question, the objective of this study is to analyze the characteristics of the scientific research that approach the fragment of literature related to reverse logistics under the perspective of performance evaluation. This analysis is guided by the purpose of generating knowledge on the studied subject and to point out possibilities of future contributions to this area. In order to guide the defined objective, the methodology carried out includes the selection of a bibliographic portfolio (BP) on the subject and a critical analysis of the articles of the selected BP, based on the theoretical constructs of a performance evaluation, in order to identify future research opportunities. To systematize this methodology, we used the intervention tool Knowledge Development Process-Constructivist (ProKnow-C) [11,12].

This research is justified by the analysis of the fragment of the selected literature, from a constructivist perspective of performance evaluation, to encompass substantial advances in this area of knowledge and in the improvement of reverse logistics performance management methods. In the literature, no substantial works that were willing to analyze research opportunities for the fragment of the scientific literature on the performance evaluation of reverse logistics were found, which validates the originality of this study. Finally, its feasibility is conferred by access to scientific articles in the CAPES journal portal (Brazilian Higher Education Personnel Improvement Coordination).

This article, in addition to this introductory section, presents in Section 2 the methodology and the instrument selected to guide the achievement of the research objective. In Section 3 is the theoretical reference, describing performance evaluation and reverse logistics from the perspective of its Performance Evaluation. Section 4 presents and discuss the results of the research in terms of the selected bibliographical, critical analysis of the BP and research opportunities identified. The final considerations will be presented in Section 5; and, finally, the article references.

2. Literature and Analyzing Methods 2.1. Methodology and Materials and Methods

This research is classified exploratory-descriptive, since it is based on the formation of a bibliographic portfolio and explores a fragment of the literature (performance evaluation of reverse logistics). The articles of the BP were extracted in the selected databases, according to the delimitations imposed by researchers in order to identify and highlight research opportunities in the academe and management fields of reverse logistics. It is descriptive as it deals with the interpretation and analysis of BP data and disclosure of research opportunities and practical challenges to promote the advancement of this area [13].

As for the technical procedures, the bibliographic research was used as it involved the analysis of articles published in the selected databases [14].

As for the data collection, they were obtained from primary and secondary sources. The primary data were obtained in a selection of the bibliographic portfolio in the Portal of CAPES Journals. In a subsequent step, the secondary data were present in the bibliometric analysis step, so that the interpreted data was obtained through the formed portfolio of articles. Thus, the approach is qualitative because the analysis steps of the articles BP were based on the knowledge, judgments, and interpretations of the authors of this research [14].

2.2. Instrument of Intervention: Knowledge Development Process-Constructivist (Proknow-C)

The Knowledge Development Process-Constructivist (Proknow-C) method is the instrument of intervention selected by the authors, since its objective is the generation of knowledge, made possible by the constructivist action demanded to operationalize its stages. The construction of knowledge is based on the researchers’ reflection that leads them to establish delimitations and make choices during a sequence of activities that make up the ProKnow-C [11,15].

The ProKnow-C consists of four steps: (i) Selection of the bibliographic portfolio (BP) of articles on the subject of research; (ii) bibliometric analysis of BP; (iii) systemic analysis of BP; and (iv) identification of the research question and objective(s) for future research [16,17,18,19,20]. Figure 1 shows the summary view of the four steps of ProKnow-C. It should be emphasized that in this article, only the steps (i), (ii), and (iv) of ProKnow-C are presented.

2.2.1. Procedures for Data Collection: Selection of the Bibliographic Portfolio

The first step proposed by ProKnow-C is with the selection of the articles that compose the bibliographic portfolio (BP) and, thus, the formation of a gross bank of articles. Based on the definition of the research axes, keywords, and their delimited combination, the articles were searched for in six selected databases available in the Portal of CAPES (Brazilian Higher Education Personnel Improvement Coordination) Journals. The search in the databases was held on 30 September and 1 October 2017. The filters delimited in each database considered only the articles published in English language, between the years of 2000 and 2017, published in journals. Figure 2 illustrates the procedures adopted from the selection of the raw articles bank to BP formation.

With the BP formed, the selected publications were analyzed with the aim of searching for opportunities for improvement to reverse logistics management.

2.2.2. Procedures for Data Analysis: Basic and Advanced Bibliometric Analysis

Continuing with the subsequent steps, Proknow-C first presents a bibliometrics analysis, which aims to identify and highlight the prominent characteristics of the analyzed BP, based on two different approaches: (i) Basic bibliometrics; and (ii) advanced bibliometrics.

For the basic variables, the researchers carried out a numerical count of occurrences and/or elements that could allow to increase their knowledge on the subject. It was evidenced that the simple numerical measurement and quantitative compilation of the data are not enough to guarantee the generation of knowledge in the researchers [12]. In this way, it is necessary to interpret the obtained data, its origin and other complementary data, to justify their occurrences. Only in this way can the researcher support his arguments and choices [11,12,21]. For this research, the following basic variables were verified: (i) Researchers with trajectory in this theme; (ii) dynamics of the authors networks formation; (iii) institutions to which authors are affiliated; and (iv) BP classification in empirical and theoretical studies.

For the next step, of the advanced variables, the analyzes were performed only in empirical articles. The analysis of the variables in empirical articles has the purpose of verifying the presence or absence of the characteristics pointed out as necessary in the theoretical researches, when the fundamentals of the performance evaluation are analyzed. Thus, the theoretical research that makes up the BP of this research are not analyzed on the advanced analysis variables.

The advanced variables used in the present research are derived from the characteristics pointed out by Neely, Gregory, and Platts [8] and by Carneiro-da-Cunha, Hourneaux, and Corrêa [22]. The features are: (i) Organization of performance measures (single, whole system); (ii) definition of performance indicators; (iii) alignment of the indicators with the strategy; (iv) purpose of the presented model (measurement, diagnosis, comparison, management); (v) interaction with the environment (internal, external, or both) [8]; (vi) focus (measurement, management); (vii) perspective (operational, strategic); (viii) scope (reduced, broad); (ix) amplitude (internal, external); (x) nature (transactional, relational); and (xi) accountability (executives, stakeholders) [22].

3. Theoretical Framework 3.1. Performance Evaluation

Organizations have undergone several changes in recent years and among them, are the form of control and measurement of the indicators. Performance evaluation should contribute to analyze and measure all deliverables of a given organizational context and not only input and output in isolation [2]. Neely et al. [8] categorizes performance evaluation as a complex activity, which consists of quantifying the efficiency and effectiveness of the actions, through a set of individual indicators, called Performance Evaluation Systems.

The evaluation system must have indicators defined internally and grouped in a set, linked to the strategic context of the company that assist in decision making, since the isolated metrics are not enough to define what should be measured [8]. Once indicators and the way in which they will be measured are defined, performance evaluation is an ongoing task and aims to increase the likelihood of an organization achieving the desired results [23].

However, several authors portray the difficulty of establishing performance metrics because of the individualities of each organization and how, even when using unique metrics for context, they should be reviewed periodically. This means that, when there is a need, new variables should be incorporated into the measurement systems [8].

The constructivist perspective of performance evaluation occurs in several environments, and also in the Brazilian context. The importance of a structured process is presented by Ensslin et al. [16] and Ensslin et al. [21], who mention that in the area of business knowledge, the performance measurement dimension provides input for management activity. Shaik and Abdul-Kader [24] show that, in reverse logistics, many indicators are raised and discussed; however, in most cases, they have no connection with the company’s strategy.

Besides not presenting a connection with the company’s strategy, many of the indicators have no relationship to each other, which creates difficulty in decision making, not allowing the organization process to be more effective and efficient. This means that the indicators are raised and measured, but they are not managed in order to achieve the organization’s goal. Ensslin et al. [21] present the importance of performance evaluation (PE) to ensure management processes are linked to the organization’s strategy. In this case, managers can make decisions based on data and secure and reliable information. In the view of more reliable processes, Carneiro-da-Cunha et al. [22] portray a new demand that arises in the area—the general tendency to have more and better transparency, control, and efficiency in the operations of organizations.

From the organization’s strategy point of view, strategic value is established by the characteristics a resource must possess in order to be considered a source of competitive advantage. Thus, strategic value depends on the ability of resources to help the corporation gain competitive advantage, reduce competitive disadvantage, or achieve other goals [25].

In conclusion, the reverse logistics sector faces the challenge of deploying its strategies into indicators with the objective of managing and creating opportunities for efficiency and operational effectiveness with a clear and focused view of the actions’ results [26].

3.2. Reverse Logistics

Rogers and Tibben-Lembke [10] defined reverse logistics (RL) as “the process of planning, application, control of operation, cost, flow of raw materials, the inventory process, finished products, the information related, from the point of consumption to the point of origin, in order to recover or create value or proper disposal”.

The holistic view detailed by Shaik and Abdul-Kader [24], defines RL as: Product return management, real-time inventory and workflow, tracking warranties, ordering and parts exchange, flow of materials and information with suppliers, data analysis, execution of repairs, customer notification, and all the logistics flow and return. Wang and Sun [27] presented distinct characteristics of RL, such as high uncertainties of supply in time, quantity and quality, and complexities of the operations, among other things.

Fleischmann et al. [28] proposed a framework based on the reverse flow of distribution from the producer to the user and back to the producer, with definite types of motivations (governance legislation, economic value, and ecological image) and types of disposal (reuse, repair, recycling, and remanufacturing), and discussed the issue of management separator counter network flows.

De Brito and Dekker [29] proposed a framework for RL that depends on five dimensions: (i) The return reasons; (ii) reception structure; (iii) the type of products and their characteristics; (iv) recovery processes and options; and (v) the actors involved and their roles. The authors looked at the characteristics of RL systems according to drivers (economics, legislation and green citizenship), reasons (manufacturing returns, distribution returns, and customer returns), recovery processes (collection, inspection, classification, selection and types of products returned, composition, deterioration, and use) and agents (to the front of the supply chain, logistics reverse, and central and specialized). Rogers, Melamed and Lembke [30] considered a number of key reverse logistics problems by modeling techniques that can be helpful in understanding problems and developing solutions, considering a high number of variables. Dowlatshahi [31] cited five strategic factors that are important for the RL: Costs, quality, customer service, environmental concerns, and political/legal concerns. Meade et al. [32] presented a framework composed of environmental factors (regulation and respect for the environment) and business factors (returns and customer satisfaction), and also observed that there were a large number of models generalized in the literature. Similarly, Rubio et al. [33] confirmed the need for further research on strategic aspects and organizational frameworks.

Companies also view the RL process at strategic, tactical, and operational levels [34]. A good RL process is necessary to deal with the return of products and to monetize the process involved in some way. Companies now recognize that the management of the RL is important and the use of performance indicators to measure its effectiveness can bring differential and competitive advantage [35]. In increasing competitive advantage, the study by Espino-Rodríguez and Rodríguez-Días [36] revealed the internal and relational capacities for value creation in consumer perception, considering essential and non-essential activities and how they relate in the generation of highest value.

Table 1 outlines the aspects related to reverse logistics based on the articles used to structure the bibliographic review. It is noticed that the authors of the BP raise several aspects related to the subject, but for the most part they converge to environmental, legal aspects, and of the operational restrictions. As an exception, De Brito and Dekke [29] takes a more holistic approach on the subject in the article: “The Framework for Reverse Logistics”. Subsequently, in the article published in 2011 by Shaik and Abdul-Kader [24], new aspects are discussed, such as tracking warranties, orders and parts replacement, and performing repairs. In this case, there is a clear evolution in the factors discussed, including new aspects as the studies and proposals evolve. Given this information, it’s validated the need for specific indicators for reverse logistics and not only the replication of indicators already known in the literature for the supply chain.

Based on the various points raised by the authors, it is evident that reverse logistics is an issue that is highly discussed regarding the management of its operations. Some corporations aim at the strategic differential, others look at marketing through trustworthiness with customers, and others are more concerned with complying with legislation. Despite having several aspects that can be seen as objectives in the operation of reverse logistics, all seek a better result, considering efficiency and effectiveness in the return of their products. 4. Results 4.1. Bibliometric Analysis: Basic Variables The first concern regarding the literature on the PE of RL consists of the identification of the most prolific authors. Prolific authors, within the framework of the constructivist perspective adopted in this research, consist of the authors who stand out in the area, not only by the number of works identified in the BP, but as effective researchers in the area during their academic trajectory. The first of the basic selected variables is to check how many BP items are from the same authors, in order to identify researchers with a track record in this area.

In Figure 3, with five articles, the most prominent author is Kannan Govindan, who is an expert researcher in the areas of operations management and sustainability at the University of Southern Denmark, with several published articles of high impact on logistics, recycling, resources, and conservation. With three articles, there are Mohamed N. Shaik and Walid Abdul-Kader, researchers of the Department of Manufacturing and Industrial Engineering of University Systems of Windsor, Canada, who have many articles on reverse logistics, warehousing, logistics performance, and management.

Following the analysis of Figure 3, having two articles in the BP, there are Ezutah Udoncy Olugu, Hamed Soleimani, Joseph Sarkis, Kuan Yew Wong, Qasim Murtaza, Rajesh K. Singh, and Saurabh Agrawal; more than authors with one item in the BP, for a total of 21 articles. Of these authors, Joseph Sarkis is highly recognized for his high impactful publications on the green supply chain (GSC), with a high volume of publications and a high contribution to the area. Ezutah Udoncy Olugu excels in publications on the green supply chain and performance and, thus, stands out with publications on the reverse logistics performance theme.

The second concern when analyzing the literature on the performance evaluation of reverse logistics consists of identifying the author articles aligned with BP references. Under the constructivist perspective adopted in this research, this process consists of measuring articles that present identification to the proposal of reverse logistics performance evaluation. When analyzed from the perspective of its authors, it allows us to verify the researchers with recognized activity in this area. Figure 4 shows the quantification of author-aligned articles in BP references.

The author most prominently identified in the references is Joseph Sarkis, with a total of 16 articles. As portrayed previously, Sarkis is currently the most active researcher in the environmental and sustainable area, being an influence for other researchers in the area. In the quantification carried out in the sequence, there is Qinghua Zhu with 14 articles, Kee-hung Lai with 10, Kannan Govindan with 7, Shad Dowlatshahi, Moritz Fleischmann, Ezutah Udoncy Olugu and Kuan Yew Wong with 3 articles and other authors, presented in Figure 5, with an article each. Zhu Qinghua and Kee-hung Lai are distinguished researchers in the GSC area, who also have important contributions in the area of sustainable logistics.

With the focus of analyzing the prominent authors of BP references, it can be seen that some of the main authors of the references, with an expressive quantity of articles, are not in the list of BP authors. One of the factors may be due to the fact that the works of these authors have been published in bases not included in the literature analyzed. In addition, other authors that appeared in the BP with only two articles, now appear with a significant number of articles in the references, as in the case of Joseph Sarkis. Kannan Govindan is the author who has great representation in the BP and also in the references of the articles, which denotes great affinity of research to the topic of the PE of RL.

Figure 5 presents a four quadrants graph of the cross between the number of articles of the authors in the BP and the number of articles of the author in the references of the BP (cross of Figure 3; Figure 4), in order to make a cut of which authors should be highlighted in the studies concerning the topic of reverse logistics performance evaluation.

From Figure 5, it is possible to identify four groups of authors referred to BP. For the distribution of the research in the quadrants, author prominence in BP and references was considered. In the upper right block, Kannan Govindan is identified with five publications in BP and seven publications in the references. The second group of authors is identified in the lower right quadrant, prominent authors in the BP such as Walid Abdur-Kader and Mohammed Shaik. In the third quadrant, prominent authors are identified in the BP references, located in the upper left quadrant. In this quadrant Joseph Sarkis, followed by Qinghua Zhu, Kee-hung Lai, and Kishore Pochampally, who are authors with significant number of publications listed in the references, but a low number of articles in the BP. In the fourth and last quadrant analyzed, in the lower left corner, we have the relevant authors for the research theme: Kuan Yew Wong, Ezutah Olugu, Moritz Fleischmann, Samir Srivastava, Ru Oqi Geng, Zelda Zabinsky, Theresa Barker, Hamed Soleimani, Rajesh Singh, Qasim Murtaza, and Saurab Agrawal.

In continuity with the analysis of the basic variables, 17 articles of the BP were identified in magazines, as shown in Figure 6.

The journal with the most articles is the Journal of Cleaner Production, followed by the Omega Magazine and the Journal of Resources, Conservation and Recycling. The Journal of Cleaner Production is one of the most recognized magazines in high impact publications in the theme of sustainable logistics, reverse logistics, and GSC. The Omega magazine presents a focus on management, which generates opportunities for publications in several areas including reverse logistics management and its performance. The Journal of Resources, Conservation and Recycling makes clear in its name that it focuses on the theme of resources, conservation, and recycling, which in turn facilitates the discussions of the theme related to the environment including reverse logistics, which is one of the focuses of the same publication.

In addition to the analysis performed of the main authors of the BP and of the references, and also of their cross-section, details of Research Collaboration Networks are presented according to Figure 7 and Figure 8.

Figure 7 shows the prominence of the authors Govindan (red cluster) and Sarkis (green cluster), with an expressive number of publications and with it the formation of a denser network of publications between them and other authors. We can see a sum of forces of knowledge, where Sarkis also publishes with renowned and recognized authors in the analysis of the BP, such as Zhu and Lai. This figure graphically shows that there is a clear collaboration network in the field of research of performance evaluation of the logistics, as well as in the themes that pervade it, such as the green supply chain, sustainable supply chain, and management. The figure also expresses another nine groups that collaborate in the research of this theme, however the publications occur in isolation.

Figure 8 presents the analysis of collaboration networks in a heatmap, that allows us to analyze the authors and networks that present the highest concentration of publications. Thus, the redder the cluster is, larger is the quantity of publications in the area of reverse logistics and stronger is the collaboration network.

4.2. Advanced Variable Analysis

In the advanced bibliometric analysis of the 21 initial articles of BP, 13 empirical articles were analyzed from the perspective of the performance evaluation bases presented by Neely, Gregory and Platts [8] and Carneiro-da-Cunha, Hourneaux JR and Corrêa [22] A preliminary analysis consisted of verifying the existence or absence of a tool, method, approach, or PE system cited in the article. From this perspective, it is verified that only one article did not present a set of measures, and it was excluded from the basis. In this way, 12 articles that present measures are the basis for the next analysis.

Figure 9 presents the first analysis of how performance measures are organized, as individual measures or a set of them. With the formation of the BP, it can be seen that of the 12 articles, 10 present joint measures of performance.

For the second analysis, presented in Figure 10, considers the relationship between PE and the environment. In this perspective, 10 articles considered this interaction, since they involved the stakeholders for strategy, the measurement of some indicator, or decision making.

The third analysis consists of verifying the alignment of the indicators with the strategy and if this is evident in the study of the analyzed article. In Figure 11, of the 12 articles, 10 of them have tools that consider an alignment indicator to strategy. It is critical that PE systems reinforce the organization’s strategy to emphasize critical elements that can influence behavior, motivate, monitor and stimulate learning [8].

The fourth variable selected analyzes whether the tool measures and compares the result with another performance or not. In Figure 12, we have the result of the analysis, that 11 articles have a process of comparison between performances.

The fifth selected variable considers the PE system interaction. Of the 12 articles, 11 present interaction with the internal and external means to the organization, and only one presents interaction only with the internal environment, as shown in Figure 13.

The sixth variable is based on the evolution of the organizational performance measurement. According to Carneiro-da-Cunha [22], performance measurement has evolved over time and is currently measured in six main topics: Focus, Perspective, Scope, Amplitude, Nature, and Accountability. In Figure 14, the results of the BP articles are summarized in these six topics.

4.3. Research Opportunities and Contributions to the Advancement of the Area From the theoretical contribution of the PE of RL, several aspects are cited in an isolated way in the field of operation control and management. The constant discussion of the theme is advancing towards the already consolidated theory of performance evaluation. However, of the role of elements that bases PE, some have not yet been addressed in the literature of reverse logistics and/or have not been applied in case studies to validate the theoretical ones.

In this way, Figure 15 aims to summarize the elements of the theoretical reference and of the bibliometric analysis that should be presented in the reverse logistics performance evaluation.

In the outer loop of the figure, six key factors that should be addressed on the studies of RL are exposed. Among the key factors found in this paper as necessary for the development of PE metrics in RL, we highlight the “process” factor in which the operational aspects of RL are included. These operational aspects were also highlighted in the work of [6] as necessary for further research. Indeed, operations are the constituent basis of RL, but need to be analyzed in their broadest possibilities and also in relation to their performance, as proposed in this paper. The other key factors listed in Figure 15 represent a breakthrough in the field of RL study by unifying important and measurable aspects for analyzing RL performance into one single framework.

The internal elements of Figure 15 present eight measurement aspects that should guide the evaluation of performance in order to satisfy the aspects of RL. It is noteworthy that many studies on RL that address the issue of strategy involved [6] in evaluating the RL practices. Based on the theory of PE, a broader view shows other aspect to be measured in RL performance, such as the satisfaction of stakeholders, also pointed out in [37] based of consumer satisfaction, the evolution of organizational learning, and the possibility of adaptation to the internal and external environments.

5. Conclusions Facing the challenges presented to reverse logistics managers in recent years, as government regulations, the pursuit for innovation and new forms of business, the environmental awareness for management of generated waste, and the increasing need for the reuse of materials, these managers have begun to seek improvement in their forms of management. At this point, the performance evaluation is configured as a tool that provides information to support management activities. However, the literature still presents a low volume of studies in this subject. The selection of the BP, performed according to the delimitations imposed by the researchers, and the methodology proposed by ProKnow-C, allowed the identification of 21 pieces of research that correspond to the searched literature: “Evaluation of Performance of Reverse Logistics”. The analysis of the BP allowed the verification of how the reverse logistics area is related to performance evaluation. Based on the basic and advanced analyses, prominent authors are highlighted as Sarkis, Zhu and Lai, who corroborate each other in publications regarding the issue of logistics management. Their publications have high impact, given by a high number of citations in the references of articles in the BP. Another prominent author is Govindan, with five articles in the BP and seven citations in references in the BP, which presents a research focus on reverse logistics performance based on mathematical models in order to establish references and control metrics.

In general, it was noted that the articles performed isolated logistics performance evaluations, which did not follow a standard of what was being measured, making it clear that there is no consolidated theoretical contribution of the indicators. The authors did not observe a structured process of PE that would serve as a subsidy to the practice of management by logistics managers. On the other hand, it can be seen that the articles which are in line with the evolution of the performance evaluation proposed by Carneiro-da-Cunha [22] and are largely aligned with Neely [8], have metric results that relate to the environment and are compared to other performances.

Finally, it should be emphasized that the most used approaches were based on specific case studies and case studies for the composition of the indicators, so that they do not follow a pattern of related metrics in order to clearly obtain the result of actions within an organization, which emphasizes a theoretical contribution gap for a reverse logistics performance evaluation model. The theoretical framework of elements that should base the PE of RL is a contribution of this paper, but has to be detailed in measurable metrics to be applied in the real case of RL. Also, the authors point out the need in future research of some empirical and quantitative research in the field of RL based on the theories that have already been developed in the academic field. This study, however, presents some limitations: (i) The search procedure for BP formation considered only the periodicals made available in the CAPES portal that were in English; (ii) only six databases were searched, so that the entire academic universe was not considered; and (iii) the work identifies studies that are references in the area, but did not find a model of PE of RL. Based on the analysis performed in the literature, the development of a structured performance evaluation tool that integrates the indicators and supports the management of reverse logistics is suggested for future investigation.

Figure 1. Stages of the Knowledge Development Process-Constructivist (Proknow-C) intervention instrument. Source: Translated from Valmorbida and Ensslin [15].

Figure 2. Procedures adopted for bibliographic portfolio (BP) selection. Source: Developed by the authors based on the procedures adopted in Proknow-C.

Figure 3. Results of articles by author for BP. Source: Research Data (2017).

Figure 4. Results of articles by authors in references. Source: Research Data (2017).

Figure 5. Crossing of the number of articles of BP authors in BP references. Source: Research Data (2017).

Figure 6. Publications by BP journals. Source: Research Data (2017).

Figure 7. Analysis of collaboration networks (the prominence of the authors Govindan and Sarkis). Source: Research Data (2017).

Figure 8. Analysis of collaboration networks (the analysis of collaboration networks in a heatmap). Source: Research Data (2017).

Figure 9. The performance evaluation system. Advanced variable 1. Source: Research Data (2017).

Figure 10. Performance evaluation and the environment. Advanced variable 2. Source: Research Data (2017).

Figure 11. Alignment of indicators with company strategy and/or objectives. Advanced variable 3. Source: Research Data (2017).

Figure 12. Evaluation and comparison of measurement tools. Advanced variable 4. Source: Research Data (2017).

Figure 13. Performance evaluation system interactions. Advanced variable 5. Source: Research Data (2017).

Figure 14. Evolution of organizational performance measurement. Advanced variable 6. Source: Research Data (2017).

Figure 15. Elements of performance evaluation for reverse logistics. Source: Prepared by the authors (2018).

Aspects of RL	[10]	[24]	[27]	[28]	[29]	[31]	[32]
Planning and management	X	X			X
Application	X
Operation control	X				X
Cost	X			X	X	X
Flow of raw materials	X	X			X
Inventories	X	X
Finished products	X
Flow of information	X	X			X
Tracking warranties		X
Orders and parts exchange		X
Data analysis		X			X
Performing repairs		X
Time supply uncertainties			X		X
Quality			X			X
Complexity of operations			X		X
Barriers			X		X
Legislation				X	X	X	X
Ecological image				X	X	X	X
3 R’s				X	X
Types and characteristics of products					X
Services						X
Customer satisfaction							X
Returns							X

Author Contributions

Conceptualization, D.V.; Data analysis, T.G.B. and M.A.M.; Formal analysis, D.V. and R.U.G.d.R.; Methodology, T.G.B. and N.C.F.; Writing, D.V. and R.U.G.d.R.; Writing-review & editing, N.C.F. and T.G.B.

Funding

This research is partially funded by CAPES (Brazilian Higher Education Personnel Improvement Coordination). and CNPQ (Brazilian National Council for Scientific and Technological Development).

Acknowledgments

Acknowledgment to the Post Graduate Program in Production Engineering at the Federal University of Santa Catarina (UFSC).

Conflicts of Interest

The authors declare no conflict of interest.

References

1. Nikolau, I.E.; Evangelinos, K.I.; Allan, S. A reverse logistics social responsibility evaluation framework based on the triple bottom line approach. J. Clean. Prod. 2011, 56, 173-184.

2. Olugu, E.U.; Wong, K.Y.; Shaharoun, A.M. development of key performance measures for the automobile green supply chain. Resour. Conserv. Recycl. 2010, 55, 567-579.

3. Fleischmann, M. Quantitative Models for Reverse Logistics: Lecture Notes in Economics and Mathematical Systems; Springer: Berlin, Germany, 2001.

4. Gungor, A.; Gupta, S.M. Issues in environmentally conscious manufacturing and product recovery: A survey. Comput. Ind. Eng. 1999, 36, 811-853.

5. Björklund, M.; Martinsen, U.; Abrahamsson, M. Performance measurements in the greening of supply chains. Supply Chain Manag. 2012, 17, 29-39.

6. Russo, I.; Confente, I.; Gligor, D.; Cobelli, N. A roadmap for applying qualitative comparative analysis in supply chain research: The reverse supply chain case. Int. J. Phys. Distrib. Logist. Manag. 2019, 49, 99-120.

7. Beamon, M.B. Measuring supply chain performance. Int. J. Oper. Prod. Manag. 1999, 19, 275-292.

8. Neely, A.; Gregory, M.; Platts, K. Performance measurement system design-A literature review and research agenda. Int. J. Oper. Prod. Manag. 1995, 15, 80-116.

9. Govindan, K.; Sarkis, J.; Palaniappan, M. An analytic network process-based multicriteria decision making model for a reverse supply chain. Int. J. Adv. Manuf. Technol. 2013, 68, 863-880.

10. Rogers, D.S.; Tibben-Lembke, R.S. Going Backwards: Reverse Logistics Trends and Practices; Logistics Executive Council: Pittsburgh, PA, USA, 1998.

11. Dutra, A.; Ripool-Feliu, V.M.R.; Fillol, A.G.; Ensslin, S.R.; Ensslin, L. The construction of knowledge from the scientific literature about the theme seaport performance evaluation. Int. J. Product. Perform. Manag. 2015, 64, 243-269.

12. Tasca, J.E.; Ensslin, L.; Ensslin, S.R.; Alves, M.B.M. An approach for selecting a theoretical framework for the evaluation of training programs. J. Eur. Ind. Train. 2010, 34, 631-655.

13. Richardson, R.J. Pesquisa Social: Métodos e Técnicas, 3rd ed.; São Paulo: Atlas, France, 1999.

14. Creswell, J.W.; Creswell, J.D. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches; Sage publications: thousand oaks, CA, USA, 2017.

15. Valmorbida, S.M.I.; Ensslin, L. Construção de conhecimento sobre avaliação de desempenho para gestão organizacional: Uma investigação nas pesquisas científicas internacionais. Rev. Contemp. Contab. 2016, 13, 123-148.

16. Ensslin, L.; Ensslin, S.R.; Pinto, H.M. Processo de investigação e análise bibliométrica: Avaliação da qualidade dos serviços bancários. Rev. Adm. Contemp. 2013, 17, 325-349.

17. Lacerda, R.T.O.; Ensslin, L.; Ensslin, S.R. Uma análise bibliométrica da literatura sobre estratégia e avaliação de desempenho. Gestão Produção 2012, 19, 59-78.

18. Rosa, F.S.; Ensslin, S.R.; Ensslin, L.; Lunkes, R.J. Management environmental disclosure: A constructivist case. Manag. Decis. 2012, 50, 1117-1136.

19. Shaik, M.; Abdul-Kader, W. Transportation in reverse logistics enterprise: A comprehensive performance measurement methodology. Prod. Plan. Control 2013, 24, 495-510.

20. Silva, R.V.; Ensslin, S.R.; Ripoll-Feliu, V.M.; Soler, C.C. E-government and public accounting information: Bibliometric and systemic analysis. Int. Res. J. Financ. Econ. 2014, 1, 76-91.

21. Ensslin, S.R.; Ensslin, L.; Imlau, J.M.; Chaves, L.C. Processo de mapeamento das publicações Científicas de um tema: Portfólio bibliográfico e análise bibliométrica sobre avaliação de desempenho de cooperativas de produção agropecuária. Rev. Econ. Sociol. Rural 2014, 52, 587-608.

22. Carneiro-Da-Cunha, J.A.; Hourneaux, F.J.; Corrêa, H.L. Evolution and chronology of the organisational performance measurement field. Int. J. Bus. Perform. Manag. 2016, 17, 223-240.

23. Lacerda, R.T.O.; Ensslin, L.; Ensslin, S.R.; Dutra, A. A Constructivist Approach to Manage Business Process as a Dynamic Capability. Knowl. Process. Manag. 2014, 21, 54-66.

24. Shaik, M.; Abdul-Kader, W. A Comprehensive Performance Measurement Framework for Reverse Logistics Enterprise. Available online: https://www.researchgate.net/publication/292836790_A_comprehensive_performance_measurement_framework_for_reverse_logistics_enterprise (accessed on 19 September 2019).

25. Espino-Rodríguez, T.F.; Gil-Padilla, A.M. The impact of outsourcing strategies on information systems capabilities in the hotel industry. Serv. Ind. J. 2007, 27, 757-777.

26. Barker, T.J.; Zabinsky, Z.B. A multicriteria decision making model for reverse logistics using analytical hierarchy process. Ómega 2010, 39, 558-573.

27. Wang, B.; Sun, L. A review of reverse logistics. Appl. Sci. 2005, 7, 16-29.

28. Fleischmann, M.; Bloemhof-Ruwaard, J.M.; Dekker, R.; van der Laan, A.E.; van Nunen, J.A.E.E.; van Wassenhove, L.N. Quantitative models for reverse logistics: A review. Eur. J. Oper. Res. 1997, 103, 1-17.

29. De Brito, M.P.; Dekker, R. A Framework for Reverse Logistics, ERIM Report Series Research in Management; Erasmus University Rotterdam: Rotterdam, The Netherlands, 2003.

30. Rogers, D.S.; Melamed, B.; Lembke, R.S. Modeling and analysis of reverse logistics. J. Bus. Logist. 2012, 33, 107-117.

31. Dowlatshahi, S. A strategic framework for the design and implementation of remanufacturing operations in reverse logistics. Int. J. Prod. Res. 2005, 43, 3455-3480.

32. Meade, L.; Sarkis, J.; Presley, A. The theory and practice of reverse logistics. Int. J. Logist. Syst. Manag. 2007, 3, 56-84.

33. Rubio, S.; Chamorro, A.; Miranda, F.J. Characteristics of the research on reverse logistics (1995-2005). Int. J. Prod. Res. 2008, 46, 1099-1120.

34. Agrawal, S.; Singh, R.K.; Murtaza, Q. Triple bottom line performance evaluation of reverse logistics. Compet. Rev. 2016, 26, 289-310.

35. Govindan, K.; Paam, P.; Abtahi, A.R. A fuzzy multi-objective optimization model for sustainable reverse logistics network design. Ecol. Indic. 2016, 67, 753-768.

36. Espino-Rodríguez, T.F.; Rodríguez-Díaz, M. Determining the core activities in the order fulfillment process: An empirical application. Bus. Process Manag. J. 2014, 20, 2-24.

37. Wang, J.J.; Chen, H.; Rogers, D.S.; Ellram, L.M.; Grawe, S.J. A bibliometric analysis of reverse logistics research (1992-2015) and opportunities for future research. Int. J. Phys. Distrib. Logist. Manag. 2017, 47, 666-687.