Insights on the Performance of Public Procurement for Water Utilities Works

Abstract

The availability of non-reimbursable funds for water supply networks has made the public procurement of construction work in this sector of particular interest in terms of performance. The purpose of this research is to identify key issues and patterns emerging from the engagement between Romanian contracting entities awarding such construction contracts and economic operators. The engagement is materialized through rounds of clarification requests during the offer preparation stage. Following the analysis of the specialized literature, it was found that the subject was not addressed in many scientific publications. The research undertaken was an empirical analysis of selected tenders. The research methodology focuses on the qualitative analysis of publicly available questions and responses, complemented by the analysis of numerical data relating to the same tenders. From the study, it results that a high percentage of all questions analyzed target the technical requirements of the contract notice documents. In some instances, the same questions are found in multiple tenders. Tender evaluation periods are lengthy. The tenders are published with a similar strategic profile (e.g., open tendering, no division into lots) and participation is dominated by bidder associations of multiple economic operators. This study concludes that the quality of the technical requirements does not support the type of strategic profile employed by the contracting entities. The rigidity of written communication makes it unsuitable to mitigate the negative effects of a flawed approach in developing technical requirements for high value, high complexity projects directly connected with Romania’s public water supply challenges.

1. Introduction

From a global to a European level, the work of many and various types of organizations reflects the importance of the global community of water resource management and the delivery of water services. Globally, the clean water and sanitation for all as the United Nations’ 6th SDG out of the 17 is telling. On the 14 July 2021, the European Commission adopted the European Green Deal, with the objective of making Europe the first climate neutral continent by 2050 [1]. Amongst the targets of the Green Deal are the provision of fresh air, clean water, healthy soil, and biodiversity.

On such a backdrop of intense interest and high expectations, Romania is the European Union (EU) member state with the lowest percentage of population connected to a water service network [2]. Further, the country is constantly failing and is challenged with financing its compliance with the EU water acquis, lagging the other member states in terms of compliance with the Urban Waste Water Treatment Directive (UWWTD) [3]. Romania benefited from 35.2 billion EUR in European structural and investment funds to spend [4] in the 2014–2020 funding period.

This research takes cognizance of one of the Organization for Economic Co-operation and Development Council’s 12 must-do principles on water governance. Specifically, it considers principle 10: Promote stakeholder engagement for informed and outcome-oriented contributions to water policy design and implementation. This research is contextualized upon the linkages between a well performing water governance, public procurement governance, public procurement practices [5] and the principles of designing and implementing effective, efficient, and inclusive water policies [6]. Hence, the public procurement practices, lessons learnt and improvements in the procurement process performance that arise from effective stakeholder engagement are connected.

The public procurement process includes mechanisms for stakeholder engagement to enable the collection of pro-active and re-active feedback, both explicit and implicit. A key stakeholder group is the for-profit economic operators in the supply chain, who are necessary for an improved water network, compliance with the EU’s UWWDT, and improved water governance.

The public sector organizations to which we refer to in this paper are defined as contracting entities, a category of public sector organizations defined at letter b, article 4 of the Romanian public procurement Law 99/2016 as entities operating in a utilities sector (water, energy, transport, postal services) [7]. These contracting entities are responsible for improving the potable and waste-water infrastructure in Romania. The contracting entities undertake tendering for water infrastructure works to align such infrastructure to the EU mandated standards, mostly financed through EU funding.

One way to describe the procurement process is to consider three stages: the planning, tender and delivery/contract implementation. The tender stage is of particular interest in this work to consider the performance of the procurement process through the lens of the engagement with the economic operators.

The planned duration of the public tender stage is one in which the performance indicators present few difficulties to establish and monitor. The duration commences on the publication date of the tender documentation. This stage includes the offer preparation period by the interested economic operators and the subsequent evaluation of offers by the contracting entities. Durations of various activities and stages are clearly and transparently regulated through legislation, as minimal, maximal, or indicative. These durations have been established to be considered practical and proven benchmarks and targets for planning and performance monitoring.

For example, in Romania, the offer preparation period for an ‘open tendering’ procedure is 30 calendar days when using electronics mediums (i.e., the Public Procurement Electronic System—SEAP) and the evaluation period is set to 60 working days [7]. In contrast, the cost and quality performance indicators for undertaking the tender stage or, for that matter, of the entire procurement process, are not rigid and explicit. They are dependent on context and the scope of the contract in terms of planned benchmarks, when such planning is undertaken.

According to Freeman, 1984 [8], stakeholders are defined as any group or individual who can affect or is affected by achieving a firm’s objectives. As such, in the public procurement context, the interested economic operators developing an offer within a procurement process are key stakeholders who can affect the achievement of a public sector organization’s objectives. Intersecting Heath’s 2006 [9] definition of engagement with the EU’s provision for rounds of clarifications during the offer preparation stage, it results that the EU allows, and even mandates, two-way, relational, give-and-takes between economic operators and public sector organizations. It also defines the measure of the effectiveness of this type of engagement: the extent to which it improves understanding among interactants, the extent to which it positively supports mutually beneficial decision making and the extent to which it fosters a fully functioning contract implementation.

Therefore, it is our intention to explore facets of the stakeholder engagement with a view to understanding the key issues and patterns impacting on the materialization of the procurement process performance indicators. In particular, this study is concerned with if and how the engagement with the economic operators in the water infrastructure sector during the offer preparation period within the tender stage of the procurement processes, as undertaken by Romanian contracting entities, leads to an improved process performance.

This paper, following the introduction, is structured as follows. First, a background section reviewing the government and professional reports followed by a review of the academic literature. Government and professional reports provide value for illustrating auditable aspects of performance in public procurement and the mandated engagement between contracting entities and economic operators. The academic literature review focuses on searching for studies that have analyzed the written clarification requests and responses undertaken during the tender stage of a public procurement process.

Second, we present the methodology employed for the literature review and for our empirical research. Third, the methodology is followed by an extensive presentation of our empirical research results. The results are separated based on the source, and more specifically, the numerical data extracted from a selection of SEAP available contract notices and the text of clarification requests and responses of the same selected contract notices.

Fourth, we discuss our findings from several perspectives to inform the final section of conclusions. In Appendix A we provide tables of the readily available and devised data categories pertaining to the empirical research, in both English and Romanian.

2. Background

2.1. Review of Literature on Public Procurement Performance

The key efficiency indicators of the public procurement process are indicated in the text of the EU public procurement directives 2014/24/EU and 2014/25/EU. These indicators are established coherently, enabling any member state, with the resources available, to achieve the intended results. The single market scoreboard tracks the various quantitative procurement process performance indicators; examples of these indicators include single bidder, no calls for bids, decision speed, Small Medium Enterprises (SME) bids, etc. Further, the harmonized Red Amber Green type benchmarks are based on qualitative policy judgements of constituted good practice; examples of these include single bidder ≤ 10%, no calls for bids ≤ 5%, decision speed ≤ 120 days, SME bids > 80%, etc. [10].

At the country level, the National Council for Solving Complaints (CNSC) [11], the body with administrative-jurisdictional activity and jurisdiction to hear appeals lodged in the procurement process, before concluding the contract [11] and the National Agency for Public Procurement (ANAP) [12], both monitor a portfolio of quantitative indicators. Examples of quantitative indicators include number of appeals, topic of appeals, number of processes, values of processes, durations of processes, competition level, etc.

In 2020, 161 contracting entities published contract notices. Out of these 161 contracting entities, 49 contracting entities (i.e., 30.43%) were active in the water infrastructure sector. It is noted that there is a significant difference in the duration in 2020, of +166 days, compared to the average duration of a tender, regardless of the type of contract targeted (i.e., 93 days) and the average duration of the tender with the ANAP ex-ante verification for the same year (i.e., 259 days). The ANAP proposes that this difference in duration is due to the ANAP’s selection methodology, which targets contracts with a high risk, indicating the complexity of the contracts, the multiple clarification questions, the appeals and the high number of offer submission durations. As an indicator for the suboptimal behavior of the contracting entities, the ‘awarding the contracts based on a single bid received’ indicator is used, which does not benefit from a qualitative assessment [13].

It is noted from the CNSC statistics that, in 2020, the percentage of appeals out of the total of contract notices published in SEAP is 7%, even before considering such filter categories of appeals related to: (a) the tender documentation; (b) EU funded tenders; (c) works contracts; and (d) works contracts for potable and waste-water infrastructure. The CNSC statistics are silent on appeals in the water infrastructure sector. Hence, there is a very narrow base of analysis for our research and thus scarce opportunities to analyze the perspective of the economic operators using CNSC as a source [14].

In 2020, 2983 appeals were registered to CNSC. Out of these, 24 appeals were prompted by a lack of clear, complete, and unambiguous response by the public sector organizations to clarification questions submitted by the economic operators. In addition, 97 appeals were prompted by perceived restrictive technical requirements. The sum of these two connected appeal categories represents 24.69% of the 490 appeals within the broader category of appeals related to the tender documentation. Nevertheless, the quality of technical requirements and the communication between economic operators and public sector organizations when clarifying written clarification questions is not addressed in the CNSC report in more detail beyond that described above [14].

The EU single market scoreboard provides a harmonized benchmark for the efficiency of the public procurement process, as sampled above. However, it is a “simplified reality” based on the data from the EU’s online version of the ‘Supplement to the Official Journal’, dedicated to the European public procurement (TED) [15], which does not cover every aspect, such as “corruption, the administrative burden and professionalism”. It is not readily apparent what the local CNSC and ANAP indicators’ performance near and longer-term country specific targets should be. The local CNSC and ANAP reports also do not make a visible a connection to the simplified EU single market scoreboard targets.

Similarly, the European’s Commission (EC) report on the implementation and best practices of national procurement policies in the internal market concludes that there are challenges in the formation of a coherent understanding of the performance of the public procurement process at the EU level. A lack of data, particularly in terms of disparities between the member states in the data collection for quantitative reporting, was one of the main insights [16].

2.2. Review of Literature on Public Procurement Engagement

As indicated in the ANAP 2020 public procurement efficiency indicator report [13], the Romanian public sector organizations exhibit a limited propensity to apply the more formal methods of engaging with economic operators. It is noted that the ANAP’s report describes market consultations as good practice in tender packaging (division into lots). Nevertheless, market consultations are not illustrated in the report, quantitatively or qualitatively. Furthermore, a close reading of the ANAP report uncovered that partnerships for innovations and procedures entailing dialogue and negotiations are limited in number.

Specifically, the frequency of engagement entailing dyadic communication is found to be limited in the water infrastructure sector. The default procedure is the ‘open tendering’, a one-stage procedure which covers the qualification of the economic operators and the evaluation of their offers in the same sole evaluation stage. This type of procedure deliberately limits engagement to the written exchange of formal documentation for clarifications both during the offer preparation period and during the evaluation period [7].

2.3. Review of Literature on Clarification Requests

Formal written engagement takes place through clarification request rounds during the offer preparation period. Such clarifications from economic operators and the subsequent response from the contracting entity are publicly available. Art. 172 alin. (2) of the Romanian Public Procurement Law 99/2016 [7], with reference to the quality of responses issued by contracting entities during the offer preparation period, solely indicates the obligation for the response to be ‘clear and complete’. Art. 108 alin. (2) of the HG 394/2016 [17] adds that the response must also be ‘unambiguous’. Further, the EC’s public procurement guidance for practitioners [18] does not address the rules of engagement with the economic operators during the offer preparation period.

A conclusion can be deduced that the country’s quantitative indicators referenced in the sections above point towards issues within the offer preparation period. Romania’s reporting under the Procurement Monitoring Report, in view of the Member States’ reporting process under the Directives 2014/23/EU, 2014/24/EU and 2014/25/EU for the period between 1 January 2018 and 31 December 2020 [19], is silent on the quality of the engagement between the public sector organizations and economic operators during this subject period. The Initiation stage, as it is framed for the reporting exercise, is limited to the moment of and immediately connected to the tender’s launch; that is, the publication of the contract notice. Further, in the Bid (offer) evaluation stage, the report recognizes and highlights the difficulties facing the evaluation committee in the offer evaluation process. The report states that it is mainly due to the management of the requests for clarifications by the committee, related to the submitted offers and associated responses from the economic operators.

In terms of the Bid submitted by the economic operator, it is regulated that any changes to the Bids as a result of clarification requests from the contracting authority/entity during the Bid evaluation stage should not entail changes to the essential aspects of the initial bid (Directive 2014/25/EU, Art. 48 alin. (6), second paragraph) [20]. However, there is no reference in the legislation to how the clarification requests submitted by the economic operators during their offers’ preparation and the subsequent responses can lead to essential changes of the tender documentation. For example, there is no reference to the consequences for or measures to be taken by contracting entities when economic operators indicate misalignments with the principles of public procurement in the clarification requests.

2.4. Review of Academic Literature

Previous research has indicated that there have been few attempts in the academic literature to capture the relevant insights pertaining to the engagement between economic operators and public sector organizations. Specifically, research on the dyadic communication which occurs during the written clarification request rounds during the offer preparation period has not been identified in the academic literature. For completeness, it is noted that no academic literature has been identified that uses the publicly available raw secondary non-numerical written data pertaining to the clarification requests and subsequent responses as source data. However, other studies which consider supplier market insights from a range of stakeholders reveal findings which invite further analysis to inform this research.

One paper explored the perspective of the economic operators as part of a study on the barriers to innovation through public procurement [21]. In 2014, Uyarra et al. used a survey of suppliers to public sector organizations in the UK to identify the procurement process innovation drivers and barriers. Among these, suppliers have pointed towards the “lack of interaction with the procuring organizations” as one of the main barriers. Similarly, at EU level, one of the risks related to the ‘response to clarification requests’ is “providing evasive answers or answers that change the specifications included in the documentation without giving business operators enough time to adapt to the submitted clarifications” [22]. Through interviews of public procurement representatives and contractors taking part in procurement procedures, Zielina [23] explores the construction sector tenders in Poland. The author notes how the questions addressed to the public sector organization by the contractors may impact the quality of the offers and the course of the proceedings.

Using Google Scholar to search for ‘clarification requests’ in the title, over a timeframe of more than 20 years, resulted in 73 papers. The resulting cohort of papers mostly explore this type of corrective input in various child interaction contexts, not public sector procurement. In the same timeframe, only one paper explores the aspect of learning in the context of public procurement, with a view of presenting the formal integration of individual and organizational learning. It does, however, emphasize the value of the learning incurred from experience as opposed to learning from sterile intellectualizing and the value of the case studies method for capturing and disseminating ‘lessons learned’ [24]. A search for ‘Romania public procurement challenges’ in the title and the same timeframe returned two results, none of which touched on the tender clarification stage written exchanges.

Finally, search results with ‘public procurement clarification requests’ anywhere in the article returned an excess of 17,000 results for the 20 years search period.

3. Materials and Methods

The method of this research is twofold: (1) a review of academic literature; and (2) an empirical study, an examination of publicly available raw secondary data, focused on the clarification requests submitted by economic operators and the subsequent responses by contracting entities of 32 contract notices.

3.1. Literature Review and Empirical Research Methodology

The intent and impact of the communications of economic operators is relevant for the type of insights sought. Therefore, analysis of authentic self-serving content created by themselves on various topics, occasions, times, contexts, was considered the most appropriate for this research. The process of conducting this research uncovered a few more objective sources for the authors to capture data and information from a key stakeholder in order to identify key issues and potential patterns, which, combined, may lead to insights on the quality of the tender documentation published by the contracting entities and on the performance of the public procurement process.

To establish a robust method of collecting and analyzing the data, a literature review was undertaken. The purpose of the literature review was to identify any attempts at analyzing this area of study. The research framework is public procurement, in the business sector of water utilities (water supply and waste-water infrastructure projects) in Romania and across Europe. We searched data bases of academic journals and publications on key words, such as ‘EU Water utilities’, ‘RO public procurement’, ‘clarification requests’, ‘public procurement effectiveness indicators’, ‘public procurement challenges’, ‘lessons learning in public procurement’, ‘learning curve in public procurement’ in the 2001–2022 interval. The searched databases were primary (SEAP tender documents, government publications, legislation) and secondary (MDPI journals, Google Scholar, Clarivate).

As illustrated above, a significant literature gap has been identified. More precisely, disparate evidence points to reasons for concern, but no in depth-analysis quantitative and/or qualitative of clarification requests and responses of any public procurement process has been identified. Therefore, an inductive approach was applied to the study.

Subsequently, the analysis of the qualitative content of the clarification requests, responses, and context, was undertaken to identify the key issues, with the intent of generating insights and reflections regarding the theoretical themes that may be proposed.

The empirical study entailed the analysis of the content of published clarification requests and responses to contract notices for water utility works. Specifically, the contract notices which include the 45232150-8 Works related to water-distribution pipelines (Rev.2) as the main Common Procurement Vocabulary (CPV) code. A total of 32 of such contract notices were published in SEAP in 2020 and had an offer submission deadline during the year 2020, as the end of the EU funding period. In most cases, the evaluation of the offers was extended into the year 2021.

The data analyzed for this study is raw secondary documentary, numerical and non-numerical, written data available publicly and extracted from SEAP. This data is publicly available, in accordance with the principle of transparency of public procurement.

The types of data used are ranked based on ease of access (from 1 being the easiest to access) and are illustrated in Table 1 below.

As these types of data pertain to the offer preparation period, the names of the economic operators who are requesting clarifications are not published with the responses. The names of the contracting entities are public, as the initiators of each of the contract notices.

The raw secondary documentary non-numeric written data is available as open-ended questions and responses, thus defined as qualitative data. For the purposes of this research, this is the most important type of data, with the other types of data (e.g., types 1 and 2 as illustrated in Table 1) providing context. The open-ended questions and responses as qualitative data have intent and impact expressed through words, requiring classification into categories for analysis. The categorization and unitizing of the data provided opportunities for quantification and graphical representation.

The data were initially analyzed through exploration, seeking to generate propositions to be tested in alignment with the existing theory.

3.2. Coding and Analysis Sequence

The coding and analysis sequence consisted of 4 steps.

First, the data were categorized and divided into 2 groups. Group 1 is comprised of readily available categories and group 2 is comprised of devised categories. The former group consists of exports and unambiguous information available in SEAP, or data categories easily calculated based on this information. The group 1 data categories are included in Appendix A, Table A1. The latter group consists of data classified from textual questions and answers of clarifications sourced from SEAP. The group 2 categories are defined by the purpose of the research. The group 2 data categories are included in Appendix A, Table A2.

Second, for two of the devised data categories, namely Question category and Targeted tender section, it was necessary to manually configure the units of data informed by the textual data available in the more than 1000 questions and responses. The units of data for Question category are included in Appendix A, Table A3. The units of data for Targeted tender section are included in Appendix A, Table A4.

Third, several charts, tables and quotations have been deployed to effectively arrange and display the classified and raw data.

Fourth, we developed and tested a hypothesis to inform the study’s conclusions. All of the data were consolidated and analyzed using the Microsoft Office Suite Excel application to ensure the traceability from the raw data to the interpreted units of data. Sections of the questions and responses which were pertinent and supported the research have been transcribed in Excel.

The following hypothesis emerged: requesting clarifications by economic operators and the receipt of responses from the contracting entities (i.e., engagement between the two parties) during the offer preparation does not lead to an improved performance of the public procurement process when tendering for water utilities works in Romania.

The testing of this hypothesis is predicated upon collecting and analyzing the empirical data inductively.

Applying the described coding process to the raw text voluminous data has simplified the complexity, enabling a more manageable and comprehensible body of data. The coding and analysis were undertaken manually, as the clarification requests and their responses consist of an excessive number of pages, numbering into hundreds of pages that are available electronically. The reference pages included scanned versions of the documents with handwritten signatures and no possibility of using software to extract the content for ease of analysis.

The focus has been on analyzing what was communicated on both sides, as this is more aligned with the objective of this research; how it has been communicated (e.g., perceived attitude, tone, etc.) is also of interest for further discourse analysis type research, however, this is beyond this paper’s scope.

4. Results

The results are presented below. First, an overview of four quantitative public procurement performance indicators pertaining to Romania, as measured by the EU and by ANAP, is given. Furthermore, the same indicators are measured within the limits of the 32 contract notices analyzed. Second, the results obtained specifically from the analysis of the 32 contract notices and the analysis of their numerical and non-numerical data are presented.

4.1. Comparison with the EU and Romania Monitored Performance Indicators

The four indicators in Table 2 are part of the set of indicators monitored by the EU [10] for all member states. The four indicators are practical in enabling a performance comparison through the analysis of the SEAP data extracted and calculated for the purposes of this research. The performance benchmarks relevant to the indicators pertain to the year 2020 [13].

4.2. Results from the Analysis of the SEAP Data

The numerical and non-numerical analysis of the content of the written clarification requests extracted from SEAP are presented in the two following sections.

4.2.1. Numerical Data Analysis Results

Overall, 32 contract notices were analyzed, pertaining to 25 works contracts. Of these, 15 notices were cancelled, pertaining to 11 works contracts. Notable repeat examples of cancellations are provided below.

A works contract was launched in 2019 on one occasion. Subsequently, the very same works contract was launched on four occasions in 2020 with increasingly higher estimated values. The works contract was cancelled all five times due to no participation. It was launched again for a sixth occasion in 2021 (31 December 2020), with an estimated value higher than the original 2019 value, by 27.82% (the equivalent of an extra 12,960,727 LEI). It was successfully awarded to the sole bidder who submitted an offer lower, 26,006 LEI below the estimated value.

A works contract was launched in 2020 on two occasions with the same estimated value and cancelled both times due to no participation. It was launched a third time, also in 2020, with an estimated value 22.97% higher than the previous value (the equivalent of an extra 13,171,199 LEI). It was successfully awarded to one of three bidders, who submitted an offer 5,758,968 LEI lower than the estimated value.

For the above two mentioned notable works contracts, the economic operators submitted a total of 44 clarification questions. For all 15 notices cancelled in 2020, the economic operators submitted a total of 270 clarification questions.

For all 32 contract notices published in 2020, the economic operators submitted a total of 1010 clarification questions.

The following findings, based on the number of clarification questions, illustrate a situation of some concern:

• Average number of clarification questions per contract notice for the 32 published contract notices is 31.56;

• Average number of clarification questions per contract for the 25 works contracts (pertaining to the 32 contract notices) is 40.40;

• Total number of clarification questions for the 17 awarded contract notices is 740;

• Average number of clarification questions per awarded contract notice is 43.52;

• Total number of clarification questions for the 15 cancelled contract notices is 270;

• Average number of clarification questions per cancelled notice is 18.

The average number of clarification questions per cancelled notice is 18 and lower than the average number of clarification questions per awarded contract notice, which was 43.52. Out of the 15 cancelled notices (Figure 1), only two had a high number of clarification questions (76 and 82). The remaining 13 had an average of eight clarification questions (270 − 82 − 76)/13. Therefore, there is a tentative correlation between the low number of clarification questions and the negative result, evidenced through the lack of participants as the main reason for cancellation. Out of the 15 cancelled notices, 11 were cancelled due to no participation.

Figure 2 illustrates the distribution of the number of clarification questions for each of the 17 contract notices that were awarded, with a total of 740 clarification questions. We found that 23.5% of the awarded contract notices (top four contract notices) accumulated 74% of the total number of clarification questions.

Figure 3 illustrates that, for all 32 published contract notices, 31.20% (top 10 notices based on the 1010 total number of clarification questions) attracted 83.96% (848 questions) of all the clarification questions.

Figure 4 illustrates that 891 clarification questions were submitted for the 21 contract notices published by the Compania de Apa Somes S.A. The 21 contract notices cover 16 works contracts; eight of these were cancelled on at least one occasion. The total estimated value of the 16 works contracts at their last launch is 947,770,979.16 LEI (194 million EUR).

Figure 5 illustrates the estimated value of the contract notices and the number of clarification questions. The conclusions drawn are that there is a weak correlation between the value of the contract notice/contract and the number of clarification questions. However, we notice a general upward trend in the number of clarification questions as the value of the contract increases. The factors influencing this trend may include the quality of the tender documentation. There are higher difficulties in preparing a good quality tender documentation for complex projects, complexity being correlated with high estimated values of the projects.

Further, it is noted that the contract notice with the highest estimated value has the highest number of clarification questions (252). This is a significant delta difference from the average of the remaining contract notices (758/31 = 24.45). It could be concluded that such a value motivates economic operators to invest resources in offer preparation, evidenced through the high number of clarification questions and through the appeals (including High Court appeals), which resulted in an evaluation period of 648 days.

Figure 6 illustrates the questions targeting the documentation and the number of questions per targeted tender section. Overall, 77.32% of the total 1010 clarification questions targeted the technical requirements on which the economic operators are expected to base their offer on.

Figure 7 illustrates the categories of the 781 questions pertaining to the technical requirements. The volume of questions tentatively indicates the failure of the technical capabilities of the contracting entities.

Figure 8 illustrates all of the question categories, comprising 1010 questions, which targeted the various tender sections.

Figure 9 illustrates that, of the 32 contract notices, 14 have had at least one request for the bid submission deadline to be extended.

Figure 10 illustrates the duration of offer preparation and decision speed for the 17 awarded contracts. In summary, there is an average of 55.5 calendar days for the offer preparation period and an average of 213.2 calendar days for the evaluation period (decision speed).

Figure 11a illustrates the offer preparation period versus decision speed per contract notice. Figure 11b illustrates the offer preparation period versus decision speed per contracting entity and contract value. In summary, for the 32 contract notices launched in 2020, there is an average of 56.34 calendar days for the offer preparation period and an average of 113.28 calendar days for the evaluation period (decision speed).

Figure 12 illustrates the total offer preparation days and number of questions for the cancelled notices. For the 15 contract notices launched and cancelled in 2020, a total of 859 days has been allocated to the offer preparation period. During this period, the economic operators submitted 270 clarification questions.

The procedure type of all 45232150-8 CPV contract notices launched and with a deadline in 2019, 2020 and 2021 was the ‘open tendering’ procedure.

A total number of 135 economic operators were involved, in some form (associate, leader), in 48 bidder associations across the 17 awarded contracts (an average of 2.77 economic operators for each association). An independent economic operator, not associated with any association, submitted an offer on only two occasions. The two examples are WTE Wassertechnik GmbH and TEHNODOMUS (WTE Wassertechnik GmbH—procedure cancelled/TEHNODOMUS—unsuccessful). Hence, for the 17 awarded contracts, the average competition is 2.88 bidders (48 associations and 1 sole bidder) or 7.94 economic operators.

At the country level, Romania shows negative values for the single bidder indicator (meaning a lower unhealthy competition, although not complemented by any further analyses/recommendations), both in the EU scoreboard and the ANAP report. Similar negative results appear based on the same single bidder indicator calculation of the data; however, with an improvement (see Table 1 above). It is important to highlight the five awarded contracts to the single bidder had a participation of 32 economic operators. Therefore, such granular analysis of the data leads to a finding with a completely different perspective. This finding serves, paradoxically, as both a contradiction and a confirmation of the single bidder quantitative indicators in the official reports.

Page 6 of the WTE Wassertchnik GmbH’s CNSC procedure result appeal (registered by the contracting entity no. 34110/19.10.2020) states: “Given the nature of the contract, its estimated value and minimum qualification requirements, demonstrating these is a challenge for medium and small companies”.

4.2.2. Non-Numerical Data Analysis Results

The following section provides groups of findings to consider in gathering insights and conclusions from the research undertaken.

GROUP 1

The first group of findings to consider are clarification requests regarding the duration of the contract, on-site visits and on the cost of water needed for the commissioning tests. These are just a sample of the many questions which induced resource deployment for clarifying simple aspects which could have been avoided through minimal planning effort on the part of the contracting entity.

Contract notice number CN1019077 includes a question to clarify the duration of the contract, with origins in several durations referenced in the Data Sheet (contract notice). Although this question was clarified by the contracting entity by invoking the impact of various components and possible configurations which determine the duration of the contract, the information was not presented clearly. This led the economic operator to submit a question about a key and basic contractual piece of information which should have been unambiguously communicated by the contracting entity. The consequences were for both parties to expend resources on reaching a shared understanding of a key and basic contractual piece of information.

Multiple clarifications were raised around the site visits to be undertaken during the offer preparation period. In total, seven of the 32 contract notices received such questions. The contracting entity appears to have lacked the ability to plan site visits and/or clearly communicate the coordination of site visits to and with the economic operators.

Compania de Apa Somes S.A. received seven questions on the cost of the water necessary for the technological tests of the network prior to handover. Three of these questions were submitted in the same contract notice. Therefore, the same question was submitted by multiple economic operators expressing interest in submitting a proposal. It is noted that this very same question is repeatedly submitted in five contract notices published at different points in time (January, February, April, June, and September 2020) (Table 3). The resulting finding is that the contracting entity neglects to address this aspect repeatedly and does not evidence a lesson learned culture of continuous improvement. The consequences are reputationally damaging to the contracting entity.

GROUP 2

Seven of the contract notices include multiple clarification questions targeting the technical requirements and bringing attention to what the economic operators perceive as conflicts with the public procurement principles. More precisely, the economic operators refer in their clarification questions to the obligations of the contracting entities to ensure equal access, to ensure and promote competition, equal treatment and not to discriminate (Table 4).

GROUP 3

From the below examples, based on the way the clarification questions are formulated, we deduce that the same economic operator notices the same requirements perceived as restrictive and repeatedly reminds the same contracting entity, on contract notices pertaining to different work contracts, of its obligation to align with public procurement principles.

The following is the same lead statement preceding the questions themselves. “[…], we request the contracting authority, […], to instruct that the scope of works to be developed in line with the principles of promoting competition, of guaranteeing equal treatment and non-discriminatory of economic operators, and in line with the principle of efficient use of public funds [art. 2, art. 155, ar. 156 of Law no. 98/2016 regarding public procurement] […]”.

CN1018030—published on the 3 January 2020

With regards to:

• The requirement to present a CE Certificate for the ductile iron pipes, pipes manufactured according to EN 545 standards, but EN is not harmonized and, consequently, does not have CE marking;

• The specific method to coat the ductile iron pipes requested (a very specific technical requirement made by the contracting entity), where the economic operator has presented arguments for the added value of using another type of coating and the industry standard which addresses the recommended coating methods.

It is worth noting that the request of the economic operator for the contracting entity to revise these requirements and others associated with it, as well as the way the economic operator has requested for the requirement to be revised, were accepted by the contracting entity. Moreover, with regards to modifying the iron pipes coating method, the contracting entity accepted the arguments of the economic operator and revised this requirement by accepting any coverage method recommended by the ISO 13470 standard as long as the pipe is protected ‘accordingly’. Such a revision from a ‘technical specification’ to a ‘performance requirement’ was not accompanied by a definition of the term ‘accordingly’.

CN1018677—published on 8 February 2020

and

CN1024406—published on the 11 September 2020

With regards to:

• The polyvinyl chloride pipe wall, initially requested by the contracting entity to be compact, and the economic operator requests to have the option to use piping with structured wall.

In both contract notices, the response of the contracting entity (the same contracting entity) was the same. The ambiguous response is constituted partially by a reproduction of the requirements already stated in the technical documentation. Further, the response referred to an adjustment of the piping’s testing requirements and had no references to the ‘structured’ wall option, which was the subject of the clarification question. Therefore, a practice of responding to questions in a way that does not clarify and/or does not address the essence of the clarification question submitted by the economic operator was noticed in this case and in others.

CN1019077—published on the 22 February 2020

With regards to:

• The requirement to present a CE Certificate for the ductile iron pipes, pipes manufactured according to EN 545 standards, but EN is not harmonized and, consequently, does not have CE marking.

The same question was submitted in the contract notice CN1018030 pertaining to another works contract launched by the same contracting entity one month previously, in January of the same year. The repetition indicates that the same technical team produced both technical documentations.

A number of other clarification questions highlighted simple errors out of a lack of attention to details, such as location references in one project pertaining to another project, thus indicating a general negligent and copy/paste practice in the technical teams that develop technical documentation for multiple projects for the same contracting entity.

GROUP 4

CN1019077—published on 22 February 2022

With regards to:

• The lack of information about the exact number of fittings necessary to execute the pipeline’s orientation, items identified by the contracting entity solely based through a financial criterion.

The clarification question is: ‘‘We believe that this percentage of approximately 5% of the estimated value of the Contract can be a defining percentage for the submitted offers, even more, by abiding by the principle of equal treatment and proportionality (Law 99/2016) all bidders should have at the tender preparation time all the necessary elements for the financial quantification of all contract components. By omitting the information indicated above, we believe that the bidders and the evaluation committee are deprived of vital information necessary to develop a unitary quantitative bid, respectively, a unitary evaluation based on equal quantities made available to the bidders’’.

The contracting entity has an obligation to describe the characteristics of the works through performance requirements, technical specifications, or a clear combination of these [7]. Describing line items by referring to their value conflicts with these legal requirements negatively impacts the procurement process. It is noted that the response to this question was clear, through which the contracting entity indicated exactly the type and quantity of the items brought into question.

With regards to:

• The request to remove the GSK certification requirement of the valves, requirement considered restrictive by the economic operator, point of view supported through referring to a CNSC 2018 decision on the same topic. GSK is an association of certain producers which represents the general and intellectual interests of its members.

The clarification question is: ‘’The CNSC decision 1329/C7/1048 of 18.05.2018 shows that, when European standards regulating the requirements of internal and external epoxydic coating, respectively, EN 14901, based on which there are existing independent organization issuing certifications, the requirement regarding the GSK certification (private body) represents a breach of art. 2, art. 155 of Law 98/2016, regarding the principles of competition, of guaranteeing equal treatment, of not discriminating economic operators and of effective use of public funds’’.

It is worth noting that the contracting entity rejected this request to exclude the restrictive GSK certification. Further, the contracting entity justified this decision by its vast and positive experience in using GSK valves as compared to non-GSK certified fittings, therefore ignoring the CNSC decision invoked by the economic operator without further explanation.

The response is: “It is accepted the use of valves which correspond to the requirements of technical sheets No. 1.3, 1.4, 2.1 and 2.3, including the requirement regarding GSK certification, this certification being based on the vast experience of the contracting authority with regards the performance of products used within the potable water supply systems, the Beneficiary having a positive experience in exploiting products with GSK accreditation compared to products which do not have this certificate. The request is not a restrictive one, it is a request based on performance criteria, confirmed through the lengthy exploitation of various types of products.” (Researchers’ emboldened text).

GROUP 5

The following examples illustrate the propagation of the same error within the tender’s technical documentation (a technical requirement considered restrictive by the economic operators), within multiple contract notices launched by the same contracting entity in different months of the year 2020.

CN1018054—published on the 7 January 2020

“[…], we request the contracting authority, […], to instruct that the scope of works to be developed in line with the principles of promoting competition, of guaranteeing equal treatment and non-discriminatory of economic operators, and in line with the principle of efficient use of public funds [art. 2, art. 155, ar. 156 of Law no. 98/2016 regarding public procurement]”.

With regards to:

• Specific requirements regarding the manufacturing method of PAFSIN joints.

It is worth noting that the request of the economic operator for the contracting entity to revise these requirements and other associated with it in the way the economic operator has requested for the requirement to be revised was accepted by the contracting entity.

The same clarification question, with minor variations, with regards to the specific manufacturing method of PAFSIN joints targets the technical documentation of the contracting entity, in the contract notices: CN1018677 (CL7), published in February 2020; CN1024406 (CL23), published in September 2020; and CN1025053 (CL17), published in October 2020.

CN1024943—published on the 2 October 2020

“The provisions of Art. 165 alin. 6) from Law no. 99/2016 which provide that ‘Technical specifications must allow all economic operators equal access to the tender procedure and must not lead to the introduction of unjustified obstacles to ensuring an effective competition amongst economic operators’ […] to allow an as possibly numerous participations of bidders […] ensuring competition […] effective use of public funds […]. The technical requirements included in the tender documentation must be prepared in line with the principles of promoting competition, of guaranteeing equal and nondiscriminatory treatment of economic operators, however the requirements highlighted above impose restrictive technical conditions for the PAFSIN joints, the effect of this measure being the elimination of those manufacturers and/or distributors of PAFSIN pipes and of related joints which do not fall under such restrictive requirements”.

It is noted that the request of the economic operator for the contracting entity to revise these requirements and other associated with it in the way the economic operator has requested for the requirement to be revised was accepted by the contracting entity.

Considering that these similar technical requirements have been highlighted as restrictive by at least two different economic operators and in multiple contract notices published by two different contracting entities in different months of the year 2020, the missed opportunities for lessons learned by these contracting entities is noticeable. Further, the propagation of errors, as illustrated by finding the same restrictive technical requirement in different technical documentations launched by different contracting entities, suggests that these documentations were prepared by the same technical team.

GROUP 6

This example is of the same clarification question submitted in two different contract notices (for two different works contracts), launched by the same contracting entity at a difference of four days.

CN1018030—published on the 3 January 2020

and

CN1018054—published on the 7 January 2020

Question: “Please confirm that only the works included in the Bill of Quantities will be executed”.

Contracting entity response: “All works necessary to ensure the functionality of the system will be executed, these works to be undertaken in accordance with the instructions of the technical specifications, the instructions of the technical project and the current legislation, and payment will be made based on the actual quantities executed during implementation, according with the contractual conditions”.

5. Discussion

5.1. From the Perspective of Reviewed Literature

The intent of this research is to explore facets of stakeholder engagement with a view to understanding the key issues and patterns impacting the materialization of the procurement process performance indicators.

The performance of the procurement process, both at EU and country level, is monitored through quantitative indicators, which are typically a result of applying mathematical formulae to the TED or SEAP data base as a first point of call extracts. The research presented in this paper focused on measuring the effectiveness of EU non-reimbursable funding of public spending aimed at increasing the percentage of the Romanian population connected to a water service network. Key benchmarks for quantitative indicators do not exist at the local level. Further, when applying qualitative analysis, it is found that the key quantitative indicators are in some cases redundant (i.e., the award criterion) and insufficient (i.e., the single bidder indicator). Critically, the revised values calculated in this research provide evidence of noncompliance by contracting entities with the public procurement legislative framework (i.e., the practice of not dividing into lots and restrictive access to SMEs).

In procuring water utility works in Romania, a decision speed of 213 days to award one works contract is high. This is in contrast with the overall performance of the Romanian public sector organizations. Additionally, when considering the use of ‘the lowest price’, it is noted that the apparent positive indicator of practice is due to the constraints applicable to the procurement processes for contracts with estimated values above the legal threshold [7]. Hence, the positive indicator is not due to proactive good practice on the part of the contracting entities.

The strategic decision by the contracting entities to not divide into lots, restricting access to SMEs, is evidenced by one of the most remarkable findings of this study, whereby 135 economic operators were linked to 48 associations as consortia bidders and only two were sole bidders. The evidence of improvements via the competition indicator when analyzing the number of bidders, consortia, or sole bidders, in water infrastructure work tenders illustrates a competition level for such contracts above the country average. However, it is paradoxically undermined by the number of actual economic operators who must pursue an association model to obtain access. There is causation between restricting the access of SMEs and obstructing the amplification of competition levels due to the decision making of contracting entities.

In previous research, encouraging bidding in consortia was deemed to be a sign of compliance with policies aimed at supporting SMEs [25]. Our research finds that, when tendering for water infrastructure works in Romania, bidding in consortia of a high number of economic operators coupled with the provision of low-quality ambiguous inconsistent technical documentation introduces friction and barriers for the SMEs and economic operators.

As initially discussed above, the strategic profile of the procurement processes show consistent themes. Apart from not dividing the high-value, complex contracts into lots, the default procedure is ‘open tendering’. A total of 10 out of the 17 awarded contracts had ‘duration of the works’ as the technical evaluation factor. Such dimensions led to the need for economic operators to form a consortium to be able to prepare an offer. This is a challenge for both SMEs and established companies (e.g., Strabag AG [26], company which participated only in two tenders, on both occasions as an associate).

Such research findings can support EU policy makers to review the monitoring of the use of EU funding. Further, the findings can support local monitoring bodies to be more vigilant and prescriptive on public sector organizations’ performance when engaging with economic operators.

We find ourselves in agreement with Melon’s 2020 recognition of the importance of market engagement through innovative public procurement procedures in order for procurement as a function to contribute effectively to the sustainability quest [27]. Further, Romania has been placed in a cluster of EU member states that are significantly behind in achieving the EU’s vision for sustainable development [28], also evidenced by the lack of green and social evaluation factors in the contract notices analyzed during this research.

Hence, we highlight the value of a more prescriptive approach to market engagement. An example of a measure to catalyze sustainable procurement is introducing a mandated form of green public procurement as an entry in the familiar ‘open tendering’ procedure [27]. It is stressed that such a tactical change, without effective market engagement and thoughtful contracting strategy, would perpetuate immature levels of sustainability performance, as uncovered by the research herein.

A similar perspective is promulgated for the integration of social agenda. A notable absence in the analyzed contract notices is the social evaluation factors. The literature review confirms that contracts with high value and long execution duration are most likely to integrate social evaluation factors, albeit with low weightings [29]. The social implications of the Romanian populations’ limited water access and the related challenges in the procurement processes are exacerbated by the absence of the water sector contracting entities’ practices of socially responsible procurement.

The engagement between public sector organizations and the market occasioned by written clarification requests in the offer preparation period is not addressed by the EU or Romanian legal framework, neither as specific rules of engagement nor monitored. We have found that this vacuum of governance enables an imbalance of power and allows flexibility to the public sector organizations to the point that it exhibits as a form of abuse of power. Lennerfors’ 2007 [30] reference to corruption is not traditional (i.e., corruption for private gain), but as deviations from pristine standards. We have seen these deviations presented through perceived conflicts with the principles of public procurement, unjustified restrictive technical requirements, and disregard for the legal obligations.

Previous findings mentioned in the literature review on the lack of interaction with public sector organizations as a barrier to innovation are confirmed. It is evidenced by the engagement being limited to written correspondence, a trait of the traditional ‘open tendering’. In addition, it is evidenced by the difficulty of contracting entities to plan and clearly communicate the details of site visits during the offer preparation period. Many of the responses to the clarification questions are indeed evasive. In contrast, requests for extensions of the bid submission deadlines, albeit ubiquitous and eventually proving nevertheless insufficient to mitigate a negative result, are always accepted. This is evidence of the tacit agreement on the high complexity of the tenders. However, cognizance of complexity is not readily apparent in any other procurement strategy decisions explored in this research.

5.2. From the Perspective of the Working Hypothesis

The hypothesis resulting from the study of existing literature was that economic operations submitting clarifications and the receipt of responses from the contracting entities (i.e., engagement between the two parties) during offer preparation does not lead to the improved performance of the public procurement process when tendering for water infrastructure works in Romania.

We have illustrated in the section above how certain established performance indicators present even more negative values than the EU and Romanian averages when narrowing down the monitoring population to processes in the water utilities sector. In addition, the findings of several themes that have not previously been considered in the same monitoring population contribute to the negative findings related to the performance of the procurement processes.

A key finding is the voluminous quantity of questions submitted by the economic operators. Without discounting the effort of the contracting entities to respond to each of these questions, it nevertheless illustrates that there is considerable interest from the market for winning public water infrastructure works contracts to maintain the persistence in repeatedly submitting clarification questions related to tender documents, which are considered by the economic operators to be flawed.

With 77.32% of questions targeting the technical requirements, we conclude that it is of suboptimal quality, conflating with a previously found barrier to procurement process innovations [21]. The technical documentation is over-specified and exhibits frequently irrelevant, incorrect, and unapplicable elements. The onus of responsibility is put on the economic operators not only for the implementation of the future works contract, but also for the preparation of their offer based on this suboptimal quality documentation.

The separation of the design stage from the construction stage is a traditional form of completing works in the execution of a project. Hence, there is a dependency of a qualitative offer for the construction stage on the quality of the technical documentation resulting from the design stage [31]. The contracting entities typically outsource the design services. The deliverables are content-heavy technical documentations, with hundreds of pages, and are difficult to analyze and evaluate in terms of quality by the client contracting entity. Sporrong 2011 [32] explores extensively the selection context of such design consultants, as a stakeholder group with a strong influence on the quality and cost of built facilities. Fazekas et al. 2013 [33] draw similar conclusions that public buyers have a limited ability to evaluate the competence of architectural and engineering consultants. There is a lack of understanding of the difference between selecting and assessing the competence of a design firm and selecting and assessing the competence of the individual consultant [33].

A combination of failures by the contracting entity and consultants results in suboptimal, reoccurring documentation. It is reasonable to assume that the same errors in different technical documentation pertaining to different contracting entities entail the use of the same pool of external consultants. Therefore, the responsibility for the quality of the technical documentation is on the external consultant, while the lack of clarity of the definition of ‘similar experience’, offer presentation requirements, contractual clauses, typographic errors, and site visits confirms a previously emerged suppliers’ perspective [21] associated with failures of the procurement teams of the contracting entities themselves. However, the contracting entity, and not the external consultants, is accountable for the overall effectiveness, or lack thereof, of the procurement process [34].

5.3. Limitations

The data analysis highlighted examples of flaws in the content of the technical documentation. It is nevertheless possible that other technical aspects have been highlighted by the economic operators as incorrect, incomplete, restrictive, etc. An exhaustive and correct analysis of more than 1000 questions and answers requires extensive technical expertise not comprehensively available within the research team, as well as the availability of native, editable manipulative formats. In addition, the applied SEAP CPV code filter resulted in a limited list of six water utilities companies, with one dominating in terms of the number of published contract notices, the number of clarification questions received, cancellations, and total contract estimated values. Nevertheless, the ANAP 2020 report indicated 49 contracting entities active in the water sector, each with a coverage and significant impact in their respective regions. Therefore, even one of them has a major influence in driving the progress in terms of water network coverage and the quality of the life of the population.

6. Conclusions

This study formulated a hypothesis that the request for clarifications by economic operators and the receipt of responses from the contracting entities (i.e., engagement between the two parties) during offer preparation does not lead to the improved performance of the public procurement process when tendering for water utility works in Romania.

Overall, we discovered that our hypothesis was simplistic, and a binary response is insufficient to characterize the complexity of the findings. It is not enough to conclude that the type of engagement we analyzed does not lead to the improved performance of the public procurement process when tendering for water infrastructure works. Indeed, we have a strong reason to conclude that it is not an effective engagement, as it did not lead to an improved understanding among interactants. It did not positively support mutually beneficial decision making.

However, we associate the ineffectiveness of the engagement and the ensuing suboptimal performance of the procurement process with an absence or ineffective contracting strategy. In the analyzed procurement processes, the decision-making is grounded firstly in a culture of established and familiar standards. A clear example is the exclusive use of the ‘open tendering’ procedure despite evidence of scope complexity (e.g., cancellations, submission extensions, lengthy evaluations). The negative consequences of not aligning the strategic profile of the procurement process with the profile of the scope and that of the market are exacerbated by the suboptimal quality of the technical documentation. Furthermore, lessons learned do not appear to inform continuous improvement, and errors are repeated.

We conclude that the procurement processes are conducted more as an administrative exercise, regardless of the results, rather than with a mindset to fulfill the duty of care of the contracting entities undertaking procurement by including considerations of social and economic effectiveness.

Therefore, a written exchange of hundreds of clarification questions and subsequent responses in one or a maximum of two rounds cannot mitigate the risks posed by a complex scope, suboptimal quality technical documentation. Finally, we have illustrated that written correspondence between the contracting entities and economic operators was ineffective, lacking good governance and exhibited an imbalance of power to the detriment of the participating economic operators and, indeed, the population of Romania.

7. Future Research

We encourage scholars from the EU member states to build upon the analysis of the qualitative nature of communications between public sector organizations and economic operators during the offer preparation period. Specifically, to explore how a qualitative or less qualitative dialogue relates to the qualitative or less qualitative implementation of the contract. It would be advantageous to better understand the rationale and sources of the breakdown in dialogue to identify pragmatic and sustainable solutions.

At the public procurement policy level, it would be beneficial to explore governance models on the rules of engagement applicable to all procedures and especially during the written exchanges associated with certain procedures.

Author Contributions

The proposed study is part of research carried out by all the authors, within a synergic collaboration, with continuous reciprocal feedback during the literature research, the writing of the text and the results obtained. All authors contributed to all phases of the work. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data supporting reported results can be found at www.seap.ro, www.anap.gov.ro, www.cnsc.ro, www.europa.eu data bases.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Cancelled contract notices and motive of cancellation.

Figure 2. Clarification questions per awarded contract notices.

Figure 3. Clarification questions for all 32 published contract notices.

Figure 4. Clarification questions submitted to Compania de Apa Somes S.A.

Figure 5. Estimated value of the contract notice and the number of clarification questions.

Figure 6. Number of questions per targeted tender section.

Figure 7. Questions targeting the technical requirements.

Figure 8. Question categories across all tender sections.

Figure 9. Number of requests to extend the submission deadline illustrated by contract notice.

Figure 10. Duration of offer preparation and decision speed.

Figure 11. (a) Offer period versus decision speed per contract notice; (b) Offer period versus decision speed per contracting entity and contract value.

Figure 12. Total of offer preparation days and questions for the cancelled notices.

Table 1

Types of data used ranked based on ease of access.

Type of Data	Access
1. The standard contract notice fields (e.g., number, publication date, name of procedure, name of contracting entity, value, status, etc.).	Available for download as a SEAP excel report export for multiple contract notices based on selected filtering criteria.
2. Other process related information (e.g., offer preparation period, offer evaluation period, total tender duration, reason for cancellation, participants/economic operators who have submitted offers, number of questions asked by the economic operators).	Available either as a calculated field from the base SEAP excel report export fields or as information embedded in subsequent SEAP pages and other locations for each of the individual contract notices of interest.
3. The clarification questions submitted by the economic operators and subsequent responses by the contracting entities.	Available for download for each of the individual subject contract notices.

Table 2

EU and Romania monitored performance indicators.

Indicator	RO General TED 2020	RO ANAPReport 2020	SEAP Export 2020(the 32 Notices)	EU Benchmark
				Green	Red
A. Single bidder	41% 1	36%	29.41% 2	≤10%	>20%
B. Award criteria(contracts awarded based solely on the lowest price)	93%	77.98% 3	0%	≤80%	>80%
C. Decision speed	89 days	65 days 4	213 days 5	≤120 days	>120 days
D. Procedures divided into lots	52%	25% 6	0	≤40%	>25%

¹ 8.5% (2060) are works contracts awarded on the basis of one bid. ² Out of the 17 contract notices with CPV code 45232150-8 awarded, 29,41% have one bidder. ³ 76.89% out of all construction contracts with the 45 CPV code. ⁴ 122 days for works contracts. ⁵ Average duration. ⁶ 11% for works contracts.

Table 3

Questions on the cost of test water.

Contract Notice	Publication Date/Time
CN1024406	11.09.2020 01:35:34
CN1022245	30.06.2020 01:35:35
CN1022245	30.06.2020 01:35:35
CN1022245	30.06.2020 01:35:35
CN1020358	11.04.2020 01:35:16
CN1019077	22.02.2020 01:37:53
CN1018031	03.01.2020 01:35:50

Table 4

Contract notices with questions mentioning conflicts with public procurement principles.

Contract Notice	Publication Date	Contracting Entity
CN1025053	07.10.2020 01:35:17	Compania de Apa Somes S.A.
CN1024943	02.10.2020 01:37:24	Compania de Apa (Buzau)
CN1024406	11.09.2020 01:35:34	Compania de Apa Somes S.A.
CN1019077	22.02.2020 01:37:53	Compania de Apa Somes S.A.
CN1018677	08.02.2020 01:36:24	Compania de Apa Somes S.A.
CN1018054	07.01.2020 01:35:00	Compania de Apa Somes S.A.
CN1018030	03.01.2020 01:35:40	Compania de Apa Somes S.A.

Appendix A

Table A1

Data categories readily available.

English	Romanian
Contract notice number	Numar anunt
Publication date	Data publicare
Errata	Erata
Contract name	Denumire contract
Procedure type	Tip procedura
Contract type	Tipul contractului
Status_Online procedure phase on 4 June 2022	Medium
Contract awarding manner	Modalitatea de atribuire
CPV code	Cod CPV
Contracting entity	Autoritate contractanta
Receipt deadline	Data limita de depunere
Estimated value	Valoarea estimate
Appeal	Contestatie
Lots	Lotizare
Cancellation motive (where applicable)	Motiv anulare (unde e cazul)
Offer submission period	Durata perioada pregatire oferta
Decision speed	Durata perioada evaluare
Participants	Participanti
Participant type	Calitate participant
Outcome for the participant	Rezultat pentru participant
Award notice date (where applicable)	Data anuntului de atribuire (unde e cazul)

Table A2

Data categories devised.

English	Romanian
Total number of questions per contract notice (for all clarification rounds)	Numar total de intrebari per anunt
Clarification round (clarifications issued proactively by the contracting entities have been ignored)	Runda de clarificari
Question # within the clarification round	Numarul intrebarii in cadrul rundei de clarificari
Question (short text extract, where possible and necessary for interpretation)	Intrebare/solicitare clarificari
Question category	Categoria intrebarii
Targeted tender section	Sectiune anunt vizata

Table A3

Units of data for Question category.

English	Romanian
Administrative	Administrativa
Qualification criterion—similar experience—unclear	Cerinta de calificare—experienta similara—neclara
Unclear offer presentation instructions	Cerinta prezentare oferta neclara
Duplicated technical requirement	Cerinta tehnica dublata
Technical requirement not applicable	Cerinta tehnica neaplicabila
Unclear technical requirement	Cerinta tehnica neclara
Restrictive technical requirement	Cerinta tehnica restrictiva
Clarification question referring to another contract notice	Intrebarea se refera la un alt anunt
Contradictory financial proposal instructions	Contradictie cerinte prezentare oferta financiara
Contradictory technical requirements	Contradictie in cerintele tehnice
Unclear duration	Durate neclare
Inattention to details	Eroare de neatentie
Incorrect offer submission forms provided	Formulare incorecte
Absence of financial information	Informatie financiara lipsa
Unclear financial information	Informatie financiara neclara
Unapplicable information	Informatie neaplicabila
Incorrect technical information	Informatie tehnica incorecta
Absence of technical information	Informatie tehnica lipsa
Unclear technical information	Informatie tehnica neclara
Absence of site visit information	Informatii vizita amplasament lipsa
Modification of technical specification	Modificare specificatie tehnica
Process	Proces
One or more unclear contractual clauses	Neclaritati pe una sau mai multe prevederi contractuale

Table A4

Units of data for Targeted tender section.

English	Romanian
Technical requirements(including technical drawings, bills of quantities, technical sheets, technical descriptions)	Cerinte tehnice(incluzand desenele tehnice, liste de cantitati, fise tehnice, caiet de sarcini)
Contract	Contract
Submission forms	Formulare

References

1. A European Green Deal. Available online: https://tinyurl.com/bdztvsxj (accessed on 31 October 2021).

2. Europe’s Water in Figures. Available online: https://tinyurl.com/txe2au5p (accessed on 31 October 2021).

3. Financing Water Supply-Sanitation-and-Flood-Protection-Romania-Workshop. Available online: https://tinyurl.com/2f2yvvyj (accessed on 31 October 2021).

4. Open Data Portal for the European Structural Investment Funds—European Commission Data European Structural and Investment Funds. Available online: https://cohesiondata.ec.europa.eu/countries/RO (accessed on 31 October 2021).

5. Public Procurement Toolbox—Organisation for Economic Co-Operation and Development. Available online: https://www.oecd.org/governance/procurement/toolbox/ (accessed on 13 December 2021).

6. The OECD Principles on Water Governance—OECD. Available online: https://www.oecd.org/governance/oecd-principles-on-water-governance.htm?_ga=2.30378682.40803334.1635703016-665222878.1635702986 (accessed on 31 October 2021).

7. Romanian Public Procurement Law 99 19/05/2016. Available online: https://legislatie.just.ro/Public/DetaliiDocument/178661 (accessed on 18 October 2022).

8. Freeman, R.E. Strategic Management: A Stakeholder Approach; Cambridge University Press: Cambridge, UK, 2010; ISBN 978/0/521/15174/0

9. Heath, R.L. Onward into More Fog: Thoughts on Public Relations’ Research Directions. J. Public Relat. Res.; 2010; 18, pp. 93-144. [DOI: https://dx.doi.org/10.1207/s1532754xjprr1802_2]

10. Public Procurement Single Market Scoreboard. Available online: https://single-market-scoreboard.ec.europa.eu/policy_areas/public-procurement_en (accessed on 1 May 2022).

11. CNSC—National Council for Solving Complaints. Available online: http://www.Cnsc.Ro/ (accessed on 31 October 2021).

12. ANAP. Romanian Government. Available online: www.anap.gov.ro (accessed on 31 October 2021).

13. ANAP. Indicatori-Eficienta-SAP-Contracte_AN2020. Available online: https://tinyurl.com/bdhzmnsz (accessed on 31 October 2021).

14. Raport 2020 RO CNSC. Available online: http://www.cnsc.ro/wp-content/uploads/2021/05/Raport.2020.RO_.pdf (accessed on 31 October 2021).

15. ENotices—Forms for Public Procurement. Available online: https://enotices.ted.europa.eu/changeLanguage?language=en (accessed on 27 November 2022).

16. EC Implementation and Best Practices of National Procurement Policies in the Internal Market. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=COM:2021:245:FIN&rid=3 (accessed on 31 October 2021).

17. HG 394 02/06/2016—Portal Legislativ. Available online: https://legislatie.just.ro/Public/DetaliiDocument/178987 (accessed on 18 October 2022).

18. Public Procurement Guidance for Practitioners. 136. Available online: https://op.europa.eu/en/publication-detail/-/publication/1f1c8329-c2c8-11e8-9424-01aa75ed71a1/ (accessed on 18 October 2022).

19. IE Procurement Monitoring Report. Available online: https://tinyurl.com/bdfn3y59 (accessed on 30 October 2022).

20. Directive 2014/25/EU of the European Parliament and of the Council of 26 February 2014 on Procurement by Entities Operating in the Water, Energy, Transport and Postal Services Sectors and Repealing Directive 2004/17/EC Text with EEA Relevance. Off. J. Eur. Union; 2014; 94, pp. 243-374.

21. Uyarraa, E.; Edlera, J.; Garcia-Estevez, J.; Georghiou, L.; Yeowa, J. Barriers to Innovation through Public Procurement: A Supplier Perspective Elsevier Enhanced Reader. Available online: https://tinyurl.com/2p87wfd7 (accessed on 2 October 2021).

22. Manea, I.; Popa, I. Risk Management in Public Procurement Process. SCECO; 2010; [DOI: https://dx.doi.org/10.29358/sceco.v0i15.145]

23. Radziszewska-Zielina, E. Public Procurement Procedure in Construction in Poland and Problems with Its Application. Organ. Technol. Manag. Constr. Int. J.; 2011; 3, pp. 269-275. [DOI: https://dx.doi.org/10.5592/otmcj.2011.1.7]

24. Snider, K.F. Integrating Individual and Organizational Learning: Agency Lessons Learned Systems and the Case Method in Teaching Public Procurement. Proceedings of the International Public Procurement Conference Proceedings; Amsterdam, The Nethrelands, 28–30 August 2008.

25. Flynn, A.; Davis, P. The Policy—Practice Divide and SME-Friendly Public Procurement. Environ. Plan. C Politics Space; 2016; 34, pp. 559-578. [DOI: https://dx.doi.org/10.1177/0263774X15614667]

26. Agenda Constructiilor-STRABAG: Portofoliu de Contracte de 330 Milioane Euro, La Nivel National Antreprenori & Dezvoltatori. Available online: https://www.agendaconstructiilor.ro/files/antreprenori-dezvoltatori/strabag-portofoliu-de-contracte-de-330-milioane-euro-la-nivel-national.html (accessed on 29 October 2022).

27. Mélon, L. More than a Nudge? Arguments and Tools for Mandating Green Public Procurement in the EU. Sustainability; 2020; 12, 988. [DOI: https://dx.doi.org/10.3390/su12030988]

28. Montalbán-Domingo, L.; Aguilar-Morocho, M.; García-Segura, T.; Pellicer, E. Study of Social and Environmental Needs for the Selection of Sustainable Criteria in the Procurement of Public Works. Sustainability; 2020; 12, 7756. [DOI: https://dx.doi.org/10.3390/su12187756]

29. Fuentes-Bargues, J.L.; Bastante-Ceca, M.J.; Ferrer-Gisbert, P.S.; González-Cruz, M.C. Analysis of the Situation of Social Public Procurement of Works at the Valencian Region (Spain). Sustainability; 2021; 13, 175. [DOI: https://dx.doi.org/10.3390/su13010175]

30. Lennerfors, T.T. The Transformation of Transparency—On the Act on Public Procurement and the Right to Appeal in the Context of the War on Corruption. J. Bus. Ethics; 2007; 73, pp. 381-390. [DOI: https://dx.doi.org/10.1007/s10551-006-9213-3]

31. Errors in the Preparation of Design Documentation in Public Procurement in Poland Elsevier Enhanced Reader. Available online: https://tinyurl.com/33afz77s (accessed on 29 October 2022).

32. Sporrong, J. Criteria in Consultant Selection: Public Procurement of Architectural and Engineering Services. Australas. J. Constr. Econ. Build.; 2011; 11, pp. 59-76. [DOI: https://dx.doi.org/10.5130/AJCEB.v11i4.2297]

33. Fazekas, M.; Tóth, I.J.; King, L.P. Corruption Manual for Beginners: “Corruption Techniques” in Public Procurement with Examples from Hungary; Social Science Research Network: Rochester, NY, USA, 2013.

34. Almquist, R.; Grossi, G.; van Helden, G.J.; Reichard, C. Public Sector Governance and Accountability. Crit. Perspect. Account.; 2013; 24, pp. 479-487. [DOI: https://dx.doi.org/10.1016/j.cpa.2012.11.005]