Measuring the Public value of e-Government: The eGEP2.0 model

Abstract

After having briefly introduced the issue of measuring e-Government vis-à-vis its impact evaluation, the paper provides an overview of the state of the art with regard to measurement of e-Government, addressing the debate on the relationship between 'public value' creation and e-Government, outlining some of the approaches advanced to measure the public value of ICT interventions in the public sector. In light of this discussion, the paper then proposes the eGEP-2.0 model which, building on its predecessor eGEP, overcome many of the limitations of existing frameworks, and more importantly pave the way for an effective impact assessment of e-Government initiatives, in relation to the policy-making process and related governance needed for their design and implementation. The results of the application of the eGEP-2.0 model on the Telematics and Informatics Plan (PiTER) of the Emilia Romagna Region in Italy are then presented and discussed. The paper concludes providing some reflections on the experience and outlining future research challenges. [PUBLICATION ABSTRACT]

Headnote

Abstract: After having briefly introduced the issue of measuring e-Government vis-à-vis its impact evaluation, the paper provides an overview of the state of the art with regard to measurement of e-Government, addressing the debate on the relationship between 'public value' creation and e-Government, outlining some of the approaches advanced to measure the public value of ICT interventions in the public sector. In light of this discussion, the paper then proposes the eGEP-2.0 model which, building on its predecessor eGEP, overcome many of the limitations of existing frameworks, and more importantly pave the way for an effective impact assessment of e-Government initiatives, in relation to the policy-making process and related governance needed for their design and implementation. The results of the application of the eGEP-2.0 model on the Telematics and Informatics Plan (PiTER) of the Emilia Romagna Region in Italy are then presented and discussed. The paper concludes providing some reflections on the experience and outlining future research challenges.

Keywords: e-government, measurement, evaluation, public value, policy-making

1. Introduction

After more than two decades of policy efforts and investments into e-Government, not only we do not have conclusive evidence on its impact (Bertot & Jeager, 2008; Castelnuovo, 2010; Savoldelli, Codagnone & Misuraca, 2012), but both the scientific and practitioners-generated literature is ridden with conceptual and terminological inconsistencies and ambiguities around concepts such as evaluation, assessment, measurement, etc. In this paper, after clearly delimiting and defining our domain of analysis, we propose a new framework to 'measure' the public value of e-Government that we call eGEP-2.0 as it represents the evolution and improvement of the eGEP1 measurement framework that one of the author of this paper elaborated for the European Commission (Codagnone et ali, 2006). We argue and show that, with respect to other available methodologies and framework, the original eGEP framework represented the most comprehensive approach available but suffered from the lack of the dynamic element needed to capture the processes that in each public agency move from initial planning, through implementation, up to monitoring and evaluating results (Misuraca & Rossel, 2011; Misuraca, G., Codagnone, C., & Rossel, P., 2012).

Before doing this, as anticipated, we need to clearly delimit the domain and the ambitions of our proposed framework as to clear the field from conceptual and terminological ambiguities that may ingenerates incorrect expectations as to what our proposal, as well as other frameworks, can produce. This requires making a clear distinction between evaluation and measurement. We do this with the support of figure 1 below.

In much of the literature addressing e-Government the two terms 'evaluation' and 'measurement' are often used ambiguously and inconsistently, with the impression that the two may be synonyms. Both evaluation and measurement consider input, output, outcomes, and impacts, but only evaluation in strict scientific sense must also looks at covariates/confounders. Strictly defined impact evaluation, in fact, aims at demonstrating that the realisation of an outcome Y can be causally attributed to the combination of input/output X (the policy treatment) and not to any other cause. On the contrary, measurement is the process by which the attributes or dimensions of a phenomenon (in this case any variable among the blocks of inputs, outputs, outcomes, and impacts) are determined and counted, as amply documented in the OECD work on public sector measurement (OECD, 2006a, 2006b, 2009). The all body of work on "Performance Measurement" or "Impact Assessment", as well as many other types of labels, can be seen as belonging together with evaluation to the 'extended' family of what we can generically refer to as "assessment", but they are clearly different from evaluation strictu sensu. Hence, neither the original eGEP framework nor other similar exercises reviewed in the next section can be and should be presented as evaluation frameworks for none of them can be used to demonstrate that the changes in a given variable of interest can be causally attributed to a given e- Government service, unless they also add an experimental or quasi-experimental component. It is important to make this clear as to avoid making claims that are not supported scientifically and empirically. The eGEP-2.0 framework we propose is therefore to be considered rigorously only an e-Government measurement framework that as such raises no claim as to demonstrating causally (i.e. evaluate) the effect that a given service or bundle of services (i.e. e-Government programme or policy) have for different constituency on a given sought outcome.

This, however, does not necessarily mean that a measurement framework such as the one we propose, cannot be linked to, and support, impact evaluation, through for instance adding a counterfactual evaluation. In fact, if the measurement is built on scientifically sound and empirically robust model of causal impact and if data are gathered on the objects of measurement steadily and reliably then eventually that can be used for a true impact evaluation. This is the object of another forthcoming paper (Codagnone, Misuraca, Savoldelli, 2014) and we will not enter into this subject here.

After having delimited clearly the domain of analysis, the paper thus provides an overview of the state of the art with regard to measurement of e-Government (Section 2), and it discusses the issue of 'public value' in relation to e-Government, outlining the debate and some of the approaches advanced to measure the public value of ICT interventions in the public sector (section 3). The paper then proposes the eGEP-2.0 as a new framework to 'measure' the public value of e-Government (Section 4), and the results of an application of the model in a real-life context is briefly presented (Section 5), before providing conclusions and some reflections on future research (Section 6).

2. A brief review of the state of the art in measuring e-Government

We have reviewed in depth the state of the art for what concerns the measurement of e-Government and the barriers deriving from its lack elsewhere (Codagnone & Undheim 2008; Misuraca et al., 2013; Savoldelli et al. 2012 and 2013). Below we extract a selective and compact summary strictly instrumental to our purpose in this paper. The first and most well-known exercises in our domain of interest have been and still are large survey based on scanning the websites of public agencies and scoring them in terms of either availability and sophistication of services' provision or of level of participation embedded in them (Capgemini, 2004; 2010; UN- DESA, 2010). These approaches have been amply criticized and most of the e-Government measurement frameworks emerged in past decades, starting from the first version of eGEP in 2005, were launched to go beyond this supply-side focus, to look at more tangible outcomes and impacts, and to be more granular (Misuraca et al., 2013).

Table 1 provides a synthetic overview of such frameworks, many of which include also user-centric measures to track take-up and satisfaction - two central parameters which allow governments to learn more about user needs and demands as well as providing a structured approach to assess policy impacts and to support the continuous improvement of eGovernment services. The frameworks or methodologies included in Table 1 have been selected using the criterion that they are some of the most cited in literature and used in practice (see Kunstelj & Vintar, 2004; Gil-Garcia & Pardo, 2005; Foley, 2006; Esteves & Rhoda, 2008)3.

In Table 1 these methodologies are assessed against two criteria: a) the dimension of public value (Carbo & Williams, 2004; Johansen, 2004; Ebrahim & Irani, 2005; Codagnone & Undheim, 2008; Heeks & Molla, 2009; OECD, 2009; Stanimirovic & Vintar, 2012) covered by the areas of impacts and/or indicators proposed; and b) the coverage of the various possible stages in the policy-making process. From this comparative analysis it emerges that current approaches are not exhaustive and comprehensive across these two criteria, which confirms the claim that the lack of a structured and comprehensive assessment framework, is among one of the key barriers delaying full adoption of e-Government (Savoldelli, Codagnone & Misuraca, 2012), and this especially for local governance (see also Anttiroiko, 2008; Belanger & Carter, 2008; Estevens & Rhoda, 2008; Kolsaker & Lee-Kelley, 2008; Kunstely & Vintar, 2009; Von Ryzin, 2009; UN-DESA, 2010; Stanimirovic & Vintar, 2012). Furthermore, the majority of such frameworks are shaped by a technology-driven approach (Dawes, 2008), under-estimating the key importance of outcome oriented approaches strictly connecting e- Government with the policy-making process (Titah & Barki, 2005; Perrin, 2006; Codagnone, & Undheim, 2008). This myopic behavior often brings eGovernment initiatives into a "lock-in/vendor-driven" situation with the consequence of risking loosing most of the expected benefits (Foley, 2006). Also important to stress the lack of covering all the relevant stages where a measurement and assessment framework would be needed, that is to say ex-ante, in-itinere, and ex-post, and the need to define a well-structured ex-ante measurement (see also Gil-Garcia&Pardo, 2005; Foley, 2006; DFP, 2012).

3. Public value and e-Government: a marriage of convenience?

If we look at the (not so young anymore) history of e-Government, it appears evident that in spite of the efforts made by many institutions and the considerable resources invested, there still not exists consensus about how to evaluate the results of the investments in e-Government initiatives. On the one hand this is due to the fact that not all the results of e-Government projects are clearly visible; and, on the other hand, the complexity itself of the concept of e-Government makes it difficult to define an evaluation framework that can be applied to all the areas covered by that concept.

In this perspective, adopting a broad definition of e-Government, such as the one suggested by the OECD (2003), where é-Government can be considered as the process of innovation of public administration in order to achieve innovative forms of government and governance through the use of Information and Communication Technologies (ICTs)', the evaluation of e-Government initiatives must be referred to its capacity of improving on the whole the performance of the organization adopting it. Thus, the concept of public value, which refers to the 'value created by government through services, law, regulations and other actions(Kelly, Mulgan & Muers, 2002) can provide an interesting support for evaluating the performances of public administrations, and thus of e-Government projects.

Public value in fact provides a broader measure than is conventionally used within the new public management literature, covering outcomes, the means used to deliver them as well as trust and legitimacy. It addresses issues such as equity, ethos and accountability. These can be considered as elements generating value also as regards the internal stakeholders involved in the management of innovation processes. Generating public value for citizens through services depends on the level of quality with which they are delivered in terms of: service availability; satisfaction levels; importance; fairness of provision; cost (Kelly, Mulgan & Muers, 2002).

The close relationship between the concept of public value and e-Government has been pointed out first of all by Kearns (2004). In a critical discussion about the excessive emphasis given to online services as the central element of e-Government, Kearns applies the work of Kelly, Mulgan and Muers directly to the evaluation of e- Government.

Public administrations in fact aims at producing value for citizens; from this perspective, the use of ICTs to improve governance is a means to improve the production of public value. Thus, an e-Government initiative resulting from a process of technological and organizational innovation can be indirectly evaluated by considering the possible increase of public value deriving from the adoption of the e-Services provided. Since e-Government aims at achieving a citizen-centred vision of government, also the evaluation of an e- Government system as regards the public value produced should be based on a citizen-centred approach (Bannister, 2002, Alford, 2002).

In this connection, the use of ICTs to improve governance, as implied by e-Government policies, can be considered as a means to increase the public value produced by public administration. Hence, the policies for e-Government can be evaluated according to their ability to increase the public administration capacity of producing public value (Kearns, 2004).

In general, a public value-based evaluation must be performed by considering the value that citizens perceive in their interactions with public administration (Alford, 2002). In this perspective, several attempts have been put forward to evaluate the public value of eGovernment. For instance, a pragmatic approach to assess the return on investment of e-Government through a public value framework has been proposed by the Centre of Technology in Government (CTG) of the University of Albany (Cresswell, A. M., Burke, G. B., and Pardo, T. A., 2006). This framework has been designed mainly as a practical tool to assist government executives and analysts in understanding and measuring the value to the public of ICT investments, including a broader view than just the financial aspects, of how e-Government can produce results of value to citizens or to the society as a whole.

Building on that approach (originally developed in collaboration with SAP), the Centre of Technology in Government (CTG) of the University of Albany has conducted a study for Microsoft Corporation and developed a renewed approach to assessing public value returns as part of an overall return on investment analysis for ICTs in government (Cresswell and Sagoyo, 2012). The approach addresses one basic question about public value assessment: What constitutes good evidence of public value impacts? The answers provided are intended to augment the return on investment analysis methods found in the 'E-Gov Economics Model: Real Impact for Better Government', developed by Microsoft. However, the approach developed is considered to have potential uses beyond connection with that model, and can be more generally useful in the assessment of public value returns to government programs and investments.

The approach consists of a way to identify, collect, and interpret a variety of evidence, both quantitative and qualitative, that can be used to assess public value. The study includes recommended methods to collect and analyse these forms of evidence and a summary of the research basis for their use. This approach is designed for being used by government practitioners and analysts in connection with return on investment (ROI) analyses using the E-Gov Economics Model to examine ICT investments by national and subnational governments. It thus has a very practice-oriented aim but it also set the basis for in-depth analysis if adopted largely and if data are made available to the scientific community.

In this perspective, recently, a review of literature conducted by Cordella and Bonina (2012), pointed out to the public value paradigm as an alternative approach for studying ICT-enabled public sector reforms. In particular, it suggests that the qualities of public sector organisations are assessed on the basis of their ability to deliver the expected value to the citizens and not only by their value-for-money ratio. However, there is still an extensive debate about how to measure outputs, outcomes and impacts in the public sector, especially in relation to the perceived public value of e-Government.

In this regard, while the eGEP framework presents some of the limitations we have mentioned above with regar dyo other measuring frameworks (see section 2), it has been largely recognized that it provides a more robust approach in assessing outcomes of e-Government initiatives (Misuraca and Rossel, 2011; Stanimirovic & Vintar, 2012). Therefore, our proposal for a new measurement framework has been built starting from the eGEP framework, which has been improved in various aspects, especially in the participation mechanisms for involving stakeholders and beneficiaries in the measurement process of e-Government services.

In this regards, as rendered in figure 2, the proposed measurement framework aims at overcoming the previous approaches, helping to establish a trust-based relationships among citizens, policy makers, civil servants and other stakeholders, so to balance the precision in the measurement of the impacts of e- Government projects, with a more transparent and participatory assessment process since the early stage of the policy-making cycle (Savoldelli, Codagnone & Misuraca, 2013).

4. eGEP-2.0: a measurement framework to assess the public value of e-Government

The simplest method to represent the consequential chain of events to be considered for an impact assessment is an Analytic Hierarchy Process (AHP) model (Saaty, 2008). From a policy-model perspective, an AHP starts from the ultimate policy goal and it is organised in several levels, linked each other with mutually dependent relationships. In literature several examples of AHP applied to policy-making process exist (Mansar, 2006; Kahraman, Demirel, & Demirel, 2007; Parra-Lopez, Groot, Carmona-Torres & Rossing, 2009; Saaty, & Vargas, 2012), however most of them adopt AHP for modelling projects' selection and/or evaluation purposes, and few of them has been used for modelling policy-making decisions. Also eGEP (Codagnone et al., 2006), that is our starting point in the development of eGEP-2.0, does the same. eGEP is structured into three levels: overall goal (level 0); outcomes criteria (Level 1); impact indicators (Level 2), and it has been applied for benchmarking the performances of e-Government web portals across administrations belonging to different EU Member States, therefore its nature mainly descriptive was successfully used in ex-post assessment of e- Government initiatives (Codagnone & Undehim, 2008). However eGEP as it was designed cannot be used to model a policy-making process as eGEP-2.0 aims to do, mainly because:

* it does not allow to represent the policy strategy formulation process, from the definition of the overall goals of a given policy planning cycle, to the definition and approval of policy guidelines (Savoldelli, Codagnone & Misuraca, 2013);

* it does not provide the necessary decisional links between the policy strategy dimension and the implementation of projects (Lundqist, 2006; Suggett, 2011);

* it does not have the ability to model the circular nature of the policy-making process (Heeks, 2006; Savoldelli, Codagnone & Misuraca, 2012).

In general terms, in fact, as described in figure 3, expected policy outcomes shape expected policy impacts (usually called policy guidelines). For achieving these guidelines, public administrations implement projects which start their lifecycle as project's ideas, by defining projects' expected outcomes, also called needs. Upon these needs, an ex-ante estimation of the degree of achievement of quantifiable objectives is provided before deciding the projects' portfolio suitable for implementing a given policy plan. These objectives are usually called expected impacts and are used for justifying project's expected outputs. Vice versa the degree of achievement of given projects' outputs, allows estimating projects' impacts and to determine to what extent projects have satisfied the need for which they have been implemented.

Of course this is a simplification of reality, where there are no deterministic cause-effect links amongst policy decisions-projects' outputs-projects/policy impacts. As anticipated (see section 1), only a cause-effect analysis based upon a robust counterfactual approach can evaluate the probabilistic and stochastic relationships that more likely represent complex realities that should be considered by policy initiatives to produce their effects (Svensson, & Pettersson-Libdom, 2008; Garbarino & Holland, 2009; Hargraves, 2010; Misuraca et al., 2013).

However the above description of the logical links between policy decisions and implementation projects allow us to organize a policy-decision model easier to be adopted in a participative process aiming to achieve consensus among stakeholders. This is also the logic through which eGEP-2.0 was designed, innovating on the original eGEP framework.

The main differences between eGEP and eGEP-2.0 are thus described in table 2, by considering four comparison criteria which are further explained below.

1) Degree of flexibility in policy planning assessment. The majority of the e-Government measurement frameworks revised in Table 1, including eGEP, are static by nature, based upon predefined criteria that are mainly related to projects that they are going to measure or compare, but rarely are designed to measure the link between policy objectives and e-Government projects (Savoldelli et al., 2013) as eGEP-2.0 does;

2) Degree of flexibility in e-Government project measurement. The eGEP model, as other models analysed is based upon a static hierarchy constituted by a fixed number of measurement criteria and indicators that do not allow addressing all the characteristics of an e-Government policy plan and its related implementation projects. eGEP-2.0 model instead is structured with a dynamic hierarchy (Schmoldt et al., 2001), that can be easily adapted to any e-Government policy plan;

3) Degree of flexibility in participative measurement through stakeholders and citizens involvement. The eGEP model, as all others, does not allow an effective participation of stakeholders and beneficiaries in the policy-making process. eGEP-2.0 instead provides the necessary support to involve all relevant stakeholders since the beginning of the policy-making process. Their involvement is foreseen in three stages (project definition and specification; weighting systems provisioning for hierarchical recomposition of evaluation criteria; weighting systems provisioning for hierarchical recomposition of policy guidelines) of the measurement process as illustrated in figure 4 and described below.

* Defining e-Government projects to be evaluated/funded. In the figure this is represented by the steps placed in the lower-left side of the diagram. The involvement of stakeholders is foreseen through the fulfilment of a web-based project's description template (Carbone, 2012), which allows to reach a consensus among various policy actors on key dimensions constituting the project's characteristics;

* Choosing indicators suitable for measurement projects in coherence with the policy guidelines (left side of the flow-diagram). This is guided by a feasibility-check procedure that allows verifying the compliancy of project's objectives with policy guidelines;

* Defining weighting system needed for hierarchical re-composition of the impact assessment model and the quantification of the e-Government adoption index. This is represented in the right side of the figure. In this case the key difference with eGEP re-composition approach consists in the adoption of weighting systems based upon stakeholders preferences and the adoption of a negotiation process amongst policy actors based upon a sensitivity analysis of the weight assigned to the various levels of the hierarchical model (Munda, 2004; Gasparotos et al., 2008; Nordstrom et al., 2012).

4) Types of impact assessment stages supported. As the majority of the frameworks analysed, also eGEP has been designed to support mainly ex-post assessment. Its applicability to other evaluation stages, even if it could be possible in theory, in practical terms is difficult, especially with regard to the estimation of ex-ante expected impacts, because it does not have enough flexibility neither in modelling the policy plan, nor in supporting participative measurement of the related implementation projects.

5. eGEP-2.0 applied to the Telematics and Informatics Plan of the Emilia-Romagna region

In this section we describe the experience conducted in the period 2011-2013 in the Emilia-Romagna Region in applying the eGEP-2.0 model as a framework to assess the Telematics Plan of the Region (namely "Piano Informatico e Telematico della Regione Emilia-Romagna - PiTER 2011-2013).

Figure 5 below presents the first three levels of the assessment framework of PiTER, based upon the eGEP-2.0 model structure described in the upper-left side of figure 4. The first three levels of the model are constructed in coherence with the objectives of the political guidelines of PiTER and in agreement with the technical and political committees that constitute the governance structure of the ICT policy implementation plan in the Emilia-Romagna Region (Carbone, 2012; PiTER, 2012). In particular, level 1 of the eGEP-2.0 model applied to the Telematics an Informatics Plan of the Emilia-Romagna Region is based upon the key policy elements of the European Digital Agenda (European Council, 2010) declined on the specific needs of the Emilia-Romagna territory. The selected criteria of level 2 of the eGEP-2.0 applied to the Plan have been based upon a consultation process involving all key actors in the Region and it has given us the possibility to design the measurement model in a transparent and participative manner. Level 3 in the model is constituted by the impact measurement indicators dynamically adapted to the characteristics of the projects that the policy plan intends to implement in order to achieve its policy objectives.

During the PiTER modelling phase, per each objective of the plan, the governance structure of PiTER defined the set of measurement criteria to be used which have then been specified in the data gathering template to help stakeholders identifying which combination of policy guideline-line objectives-criteria better matches their project idea (see figure 6).

This matching process was based upon a series of interactions amongst the project proposers and the governance structure of PiTER, creating a participatory process for the construction of the implementation plan, during the duration of this consultation process (September - December 2011) exchanging and discussing on the basis of quantitative and qualitative data about the project.

Table 3 provides some evidences of such participation for the year 2012 when eGEP-2.0 has been applied for the first time to the whole decision-making process of PiTER (2011-2013).

The application of eGEP2.0 not only has allowed to speed up the yearly ex-ante policy evaluation process of at least six months, but it has also allowed to assess 34 projects, with a degree of coverage of the Policy Plan Objectives of PiTER close to 90%, and with a correspondence rate between policy objectives and implementation projects of 1.3.

Moreover it has also allowed achieving consensus among main e-Government actors of the Emilia-Romagna region on about 100 assessment indicators, which in average terms means about 3 indicators for each policy implementation project or above 4 indicators for each policy objective, thus providing an important baseline for monitoring and impact

In the case of PiTER, eGEP-2.0 has been designed for 5 policy guidelines, underpinning 30 policy objectives and measured by 8 different assessment criteria (i.e. the e-GEP 2.0 foreseen a partial overlap of the assessment criteria across the guidelines), and up to now it has allowed to assess about 70 e-Government projects over a period of two years (2012 and 2013).

Similarly, the eGEP2.0 methodology allowed building consensus between the policy makers and stakeholders involved in e-government initiatives in the Emilia-Romagna region in order to define the final selection of the portfolio of projects aimed at implementing PiTER.

The portfolio of PiTER's projects selected in 2012 through a participatory process supported by eGEP2.0 is shown in graphical manner in Figure 7, where it is mapped against the 'ICT maturity model' defined in Savoldelli, Codagnone & Misuraca, (2012)

The 34 e-Government initiatives funded by PiTER in the year 2012 are represented in figure 7 by circles of various colours and sizes. Each colours identifies one of the 5 policy guidelines (see Figure 8 below) in which the Telematics and Informatics plan of the region has been divided (deep green - infrastructural projects; light green - typical e-Government services; light blue - inclusions actions; deep blue - innovative e-government services; red: public data management and open data initiatives), while the size of the circle represents the amount of funding allocated (the range from the maximum amount of about 17 Millions of euros of the MiMuovo project - see the biggest deep blue circle - and the minimum amount of about 100.000 euros of the project Ultranet - see the smallest deep green circle).

The distribution of the e-Government initiatives across the maturity model have been defined through a 1-10 scale aimed at measuring the citizen centricity and the process orientation of each project.

In figure 7 are reported the average position of each e-Government initiative on the two axes calculated on the basis of a consensus building process among responsible of the e-Government evaluation unit of the Emilia- Romagna region.

As can be seen in figure 7, the positioning of the e-Government initiatives of PiTER 2012 reflects a quite mature e-Government for the Region. As a matter of fact the majority of the funded projects are located in the upper part of the maturity curve, with some exception due to the completion of the broadband infrastructure (Net4all project) and the deployment of a Wifi network across the main cities in the regional territory (WiFed). This combination of projects shows also that the Region is supporting the process of open data towards citizens and enterprises (see open data initiative and GEOportale project) and a process of innovation in e- Government services aimed at improving public mobility (MiMuovo project) and intelligent and integrated monitoring systems (see GreenNet, and VideoNet).

Figure 8 below instead provides an overview of the agreed expected impact of the e-Government projects funded in 2012, according to each of the PiTER Policy Guidelines, in terms of improvement of the expected outcomes and assessed against existing baselines. More specifically, it provides an overview of the expected impacts of the 34 funded projects on the degree of achievement of the objectives of the 5 PiTER policy guidelines, where the blue line represents the baseline from which the expected impacts start to be evaluated.

As expected the most mature guideline is the deep green (GL1 - Technological access right) which objectives are the completion of the broadband infrastructure of the Region and the establishment of private-public partnerships to make sustainable their management in a long-term perspective. With the initiatives funded under PiTER 2012 the Emilia-Romagna region is expecting to achieve a degree of fulfilment of the objectives of this guideline of almost 65%.

The most innovative services are those related to GL4 - "Open Data to all" that is in line with the level of maturity of the regional territory (see also figure 7). In this case all funded project starts from a baseline close to 0, while an overall impact on this guideline of almost 25% is expected.

The weighted aggregation of the 5 impacts described in figure 8 provides the global e-Government adoption index that is expected to be achieved by the e-Government projects funded under PiTER 2012.

Finally in Figure 7 we have reported the results of the evaluation of what can be defined the "social or public Value" of the projects portfolio of PiTER 2013. This "public value" represents, in the ex-ante evaluation that we have organized with the actors of the regional policy making process of e-Governmnet, the ratio between the investments requested by each project to produce the expected outcomes and the normalized estimated value of the expected outcomes: higher is the ratio value, lower is the "public value " of the project.

* An interesting result of this analysis is related to the "value" calculated for the projects related to the ICT infrastructural investments, namely "WiFED"4 and "Ultranet"5. In the Figure 7 the Wi-Fi project shows the most important "public value" while the broadband one even if it is ranked in the first position has a less "public value". This is mainly due to the high difference of investments that the two projects have (30 million euro for the completion of the broadband in the territory of the Region, against about 200.000euro for completing the Wi-Fi coverage of most populated urban areas in the Region). According with the regional actors, this assessment indicators that has been introduced in the evaluation of PiTER 2013 in addition to the others described for PiTER 2012 is an important element for the final judgement of which projects have to be included in the investment portfolio of eGovernment in a period of dramatic economic crisis and shortages of funds available for further investment in eGovernment as the one we are living today.

6. Conclusions and future research

In this paper we have discussed the issue of measuring e-Government and analysed some of the main frameworks of measurement used in practice and at policy level for so doing. In light of the comparative analysis of selected measurement frameworks, we have discussed approaches to study the impact of e- Government in relation to public value creation, which have been proposed and discussed in literature and practice. However, while this is indeed, in our opinion, a marriage of convenience, these approaches which are based mainly on indicators defined to measure the direct or indirect impact of ICT adoption in public sector administrative and economic performances, often neglect the social and political impacts of public value creation, and how to assess them.

For this purpose we propose a new framewok of measurement, the eGEP-2.0 model, which building on its predecessor, eGEP, overcome many of the limitations of existing frameworks, and more importantly pave the way for an effective impact assessment of e-Government initiatives, in relation to the policy-making process neded for their design and implementation.

What is more important, we did not limit ourselves to propose a conceptual framework, but actually tested its validity by applying it to a real-life setting in the case of the Telematics and Informatics Plan (PiTER) of the Emilia-Romagna Region, where the application of the eGEP2.0 methodology allowed to support better the overall policy planning and evaluation process of e-Government starting with the definition of the policy strategy and the formulation of an ex-ante assessment based upon a business case for each project implementing the plan.

The findings from the 'validity test' suggest the framework is robust enough for being an initial starting point to guide the possible development of theoretical perspectives and practical applications required, on the one hand, to contribute developing a better theoretical understanding of the impacts of e-Government policy interventions, from a public value perspective; and on the other hand to implement a practical-oriented measurement instrument to assess e-Government policies and the consequences of different policy implementation options.

The proposed methodology is also a clear enhancement in respect to the existing approaches proposed so far and it is easily replicable in different policy contexts. However the proposed methodology clearly shows further areas of improvement both from a research and practice perspective.

In particular its robustness and effectiveness should be tested in multi-level government contexts and with respect to different governance mechanisms. This implies that further research is required to better understand the interrelations between the various dimensions of the eGEP2.0 model, as well as to test on a larger scale its validity and with more depth. This would require for instance to eventually setting up a social (e.g. quasi-natural) experiment to observe and monitor the changes that e-government interventions are generating in different contexts.

In this perspective, an interesting domain of research would be to explore possible intersections of the proposed methodological approach and the one aimed at establishing cause-effect relationships between policy actions and social impacts, in order to better appreciate how their synergic use in different stages of the policy planning process could be considered.

Moreover it should be investigated better how to improve the citizens' participation mechanisms in the consensus-building process underpinning formal policy-making procedures related to the various aspects of the proposed methodology. At present, in fact, the participation of citizens to the decision-making process is mediated by stakeholders and doesn't include any direct mechanism for achieving consensus around specific policy decisions.

Further research is therefore required to monitor constantly the evolution of e-Government applications and the emerging challenges they bring about from a technical, governance and public policy perspective, and at the same time, dealing with the need of policy-makers and public organisations to engage with large audiences and to respond to the ever increasing potential that user generated data made available from citizens and other stakeholders is generating.

Disclaimer: The views expressed in this paper are purely those of the authors and may not in any circumstances be regarded as stating an official position of the European Commission.

Acknowledgments

The section on the application of the methodological framework to the case of the Emilia Romagna Region has benefited of the collaboration with the evaluation unit of the ICT Direction of Emilia-Romagna region. Special thanks go to Carla Carbone, coordinator of the evaluation unit, and the Ervet SpA team, who have provided an extremely valuable effort to sustain and contribute in the design and deployment of the framework to the Emilia Romagna case. Without their competence and passion it would have not be possible to realise the concrete application and further improvement of the eGEP-2.0 methodology.

Footnote

1 The eGEP acronym stands for eGovernment Economics Project, funded by the European Commission in 2005-2006 and carried out by a consortium composed of RSO Spa and the LUISS University of Rome, under the leadership of Prof. Cristiano Codagnone.

2 Based on several sources: see among others (Algemene Rekenkamer, 2006; Boyne et al, 2003; Codagnone, 2007; Codagnene & Undheim, 2008; Hatry, 1999; Heeks, 2006; Heeks & Molla, 2009; Irani et ali, 2005; OECD, 2006a; 2006b, 2009).

3 They are: eGovernment Signpoint (Danish Digital Task Force, 2004); MAREVA (ADAE, 2007); WeBe 4.0 (Rothig, 2004; 2010); eGEP (Codagnone et al 2006); NOIE (Australian Government, 2005); GOL Performance Measurement (Treasury Board of Canada, 2004); eGovernment Satisfaction Index (Freed, 2009); VMM (Booz-Allen-Hamilton, 2004); DVAM (AGIMO, 2004); Gateway Process (DFP, 2012).

4 WiFED project aims at completing the wi-fi coverage of the most populated urban areas of the Emilia-Romagna Region

5 Ultranet project aims at completing the broad band coverage of the Emilia-Romagna territory.

References

References

ADAE, (2007), MAREVA methodology guide: Analysis of the value of ADELE projects. Fourth High Level Seminar on Measuring and Evaluating E-Government, Dubai.

AGIMO, (2004), Demand and Value Assessment Methodology - DAM & VAM. Australian Government Information Office, Commonwealth of Australia.

Algemene Rekenkamer (2006), Performance audit manual, The Hague.

Alford, J 2002, 'Defining the client in the public sector: a social exchange perspective,' Public Administration Review, vol. 62, no. 3, pp. 337-346.

Anttiroiko, A-V., (2008). Local eGovernment for Development: preliminary remarks.

Australian Government, (2005), ICT and Australian Productivity: Methodologies and Measurement. Occasional Economic Paper, Department of Communication, Information Technologies and the Arts.

Bannister, F. (2002). Citizen Centricity: A Model of IS Value in Public Administration. Electronic Journal of Information Systems Evaluation, 5(2).

Bannister, F. (2007). The curse of the benchmark: An assessment of the validity and value of e-government comparisons. International Review of Administrative Sciences, 73(2), 612-631.

Bannister, F., & Connolly, R. (2011). Transformation and public sector values. tGov 11. London: Brunel University (Brunel University).

Bertot, J. C., Jaeger, P. T. (2008). The E-Government paradox: Better customer service doesn't necessarily cost less. In Government Information Quarterly, 25(2), 149-154.

Belanger, F., Carter, L., (2008). "Trust and risk in eGovernment adoption". In Strategic Information Systems, 17( ), pp: 165- 176.

Bonina, C., & Cordella, A. (2009). Public sector reforms and the notion of 'public value': Implications for egovernment deployment. AMCIS 2009 proceedings.

Boyne, George A., Catherine Farrell, Jennifer Law, Martin Powell, and Richard M. Walker. 2003. Evaluating Public Management Reforms: Principles and Practice. (Buckingham, UK: Open University Press).

Booz-Hallen-Hamilton, (2004), An Introduction to Value Measurement Methodology. Booz- Hallen- Hamilton presentation available at http://www.fgdc.gov/policyandplanning/50states/introduction-to-vmm-bah-0ct-2004.pdf

Capgemini, (2004). Digitizing Public Service In Europe: Putting Ambition into Action, 9th Benchmark Measurement, (2010). Delivered for the European Commission, DG Information Society: Brussels, available online at: http://ec.europa.eu/information_society/newsroom/cf/document.cfm?action=display&doc_id=747

Capgemini, (2010). Digitizing Public Service In Europe: Putting Ambition into Action, 9th Benchmark Measurement, (2010). Delivered for the European Commission, DG Information Society: Brussels.

Carbo, T., & Williams, J., G. , (2004). "Models and metrics for evaluating local electronic government system and services". In Electronic Journal of e-Government. 2(2), pp: 94-104.

Carbone, C. (2012). La valutazione dei Progetti ICT: il caso del PITER 2011-2013. Paper presented in the "Bologna Smart City workshop - SmartER", October, Bologna.

Castelnovo, W. (2010). Is there an eGovernment Paradox? In D. O'Donnel (Ed.), 10th European eGovernment Conference on eGovernment (pp. 90-98). Limerick, Ireland: Academic Publishing.

Codagnone, C., Caldarelli, L., Cilli, V., Galasso, G. & Zanchi, F. (2006). Compendium to the Measurement Framework, eGEP Project. DG Information Society, Brussels. [online] Available at http: http://82.187.13.175/egep/asp/E_Home.asp.

Codagnone C. (2007). Measuring eGovernment: Reflections from eGEP Measurement Framework Experience. European Review of Political Technologies, 4, 89-106.

Codagnone, C., & Undheim, T., (2008)."Government Efficiency and Effectiveness: The Theory and Practice of Evaluation and Measurment". In European Journal of ePractice.1(4), pp: 1-15.

Cordella, A. and Bonina, C. M. (2012) A public value perspective for ICT enabled public sector reforms: a theoretical reflection. Government information quarterly, 29 (4). 512-520. ISSN 0740-624X

Cresswell, A.M., G.B. Burke, & T.A. Pardo. 2006. Advancing Return on Investment Analysis for Government IT: A Public Value Framework. August. Albany, NY: The Center for Technology in Government, the Research Foundation of State University of New York.www.ctg.albany.edu/publications/reports/advancing_roi

Cresswell a., and Sagoyo, D. S., (2012). Developing Public Vlue Metrics for Returns to Governmebt ICT Investments: A report to Microsoft Corporation, Center for Technology in Government, The Research Foudation of the State University of New York, Albany, USA

Dawes, S., S., (2008). An Exploratory Framework for Future eGovernment Research Investments. Proceedings of the 41st Hawaii International Conference on Systems Science.

Danish Digital Task Force, (2004) The Danish eGovernment Strategy 2004-2006: Realising The Vision, DTF, Copenhagen, http://e.gov. dk/uploads/media/strategy_2004_06_en_01.doc.

DFP, (2012). Dipartment of Finance and Personnel. The Gateway Review Process. Available at: http://www.dfpni.gov.uk/gateway-review-process

Ebrahim, Z., Irani, Z., (2005). "eGovernmnet adoption: architecture and barriers". In Business Process Management Journal, 11(5), p: 589-611.

Esteves, J., Rhoda, C. J., (2008). "A comprehensive framework for the assessment of eGovernmnet projects". In Government Information Quarterly, 25, pp: 118 -132.

EUPractices, (2007), European eGovernment 2005-2007: taking stock of goods practices and progress towards implementation of the i2010 eGovernment Action Plan. European Commission Report, DG Information Society and Media.

European Commission, (2005). "i2010 - A European Information Society for growth and employment". Communication from the Commission to the Council, the European Parliament, the European Economic and Social Committee and the Committee of the Regions.

European Council, (2010). European Council conclusion on Europe 2020 strategy inclusive Digital Agenda. Conclusion of the European Council.

Foley, K., (2006). Using the value measuring methodology to evaluate government initiatives. Proceedings of the 2006 Crystal Ball User Conference. May 1-3. Denver. Colorado.

Freed, L., (2009). eGovernment Satisfaction Index. ForeSee Results periodical publication.

Garbarino, S., & Holland, J., (2009), Quantitative and Qualitative Methods in Impact Evaluation and Measuring Results. Issue Paper commissioned by the UK Department for International Defelopment.

Gasparotos, A., EL-Haram, M., & Horner, M., (2008). "A critical review of reductionist appraches for assessing progress towards sustainability". In Environmental Impact Assessment Review. 28 (2008), p: 286-311.

Gil-Garcia, J., R., & Pardo, T., A., (2005). "eGovernment success factors: mapping practical tool to theoretical foundations". In Government Information Quarterly, 22(2), pp: 187-216.

Hargraves, M., B., (2010), Evaluating System Change: A Planning Guide. Mathematica Policy Research Inc.

Hatry, H. P. Performance Measurement: Getting Results. Washington, D.C.: UrbanInstitute Press, 1999.

Heeks, R., (2006). Understanding and Measuring eGovernment: International Benchmarking Studies. Paper prepared for UNDESA workshop on "eParticipation and eGovernment: Understanding the Present and Creating the Future". 27-28 July, Budapest, Hungary.

Heeks R., Molla A., (2009), Impact assessment of ICT development projects: a compendium of approaches, Manchester University Institute for development policy and management, Ed. Development Informatics Group.

Johnsen, A., (2004), The politics of performances measurement: what does 25 yrs of experiences tell us about the state of performance measurement in public management, European Group of Public Administration Conference, Lublijana.

Kahraman, C., Demirel, N. C., and Demirel, T. (2007) 'Prioritization of e-Government strategies using a SWOT-AHP analysis: the case of Turkey', European Journal of Information Systems, Vol. 16, No.3, pp. 284-298.

Kearns, I. (2004). Public value and e-government. Institute for Public Policy Research, London, UK

Kelly, G., Mulgan, G. and Muers, S. (2002), Creating Public Value: An Analytical Framework for Public Service Reform, discussion paper prepared by the Cabinet Office Strategy Unit, United Kingdom.

Kolsker, A., & Lee-Kelley, L., (2008). "Citizens'attitudes towards eGovernment and eGovernance: a UK study". In International Journal of Public Sector Management. 21(7), pp: 723-738.

Kunstelj, M., & Vintar, M., (2004). "Evaluating the progress of eGovernmnet development: A critical analysis". In Information Policy, 9(3-4). pp: 131-148.

Irani Z Love P.E.D; Elliman T; Jones S and Themistocleous M 2005. 'Evaluating e-government: learning from the experiences of two UK local authorities'. Information Systems Journal. 15:pp.61-82.

Lindquist, E.A. 2006, 'Organising for policy implementation: the emergence and role of Implementation Units in policy design and oversight', Journal of Comparative Policy Analysis, Vol 8, No. 4, December, p 421.

Mansar, S. (2006), E-Government Implementation: Impact on Business Processes, IEEE, pp.1-5.

Misuraca,G., (2012), Assessing ICT-enabled innovation for governance and policy-making, Phd thesis, College of Management of Technology, Ecole Polytechnique Fédérale de Lausanne (EPFL), Switzerland

Misuraca,G., (2012), e-Governance: Past, Present and Future: A theoretical framework for prospective policy analysis, in 'eGovernance, A Global Journey'. Vol. 4, 09/2012. (pp. 185-199). Edited by Finger, M., and Sultana, F., IOS Press, BV, Amsterdam, http://www.iospress.nl/book/e-governance-a-global-journey

Misuraca G. and Rossel, P., (2011). Reflexivity, modelling and weak signals of transformational tracks to support both micro- and macro-measuring of Information society services. In Proceedings of the 5th International Conference on Theory and Practice of Electronic Governance (ICEGOV2011), Tallinn, Estonia, 26-28 September 2011 - ACM International Conference Proceedings Series, ACM PressMisuraca, G., and Rossel, P., (2011). Measuring and Meta- measuring; in search of new Pathways for Modelling Impacts of ICT-enabled services on the Information Society, (2011). In proceedings of the IFIP 2011 eGovernment Conference (Delft, Netherland, 28 August - 2 September 2011 - http://www.egov-conference.org/egov-2011

Misuraca, G., Codagnone, C., & Rossel, P., (2013). "From Practice to Theory and back to Practice: Reflexivity in Measurement and Evaluation for Evidence-based Policy Making in the Information Society", in Government Information Quarterly, 30(2013) S68-S82 Elsevier Publishing, USA.

Munda, G., (2004). "Social multi-criteria evaluation: methodological foundations and operational consequences". In European Journal of Operational Research. 158 (2004), p: 662-667.

Nordstrom, E-M., Ohman, K., & Eriksson, L., O., (2012). "Appraches for aggregating preferences in participatory forest planning- an experimental study". In The Open Forest Science Journal. 5(2012), p: 23-32.

OECD (2010), The Economic Impact of ICT, Measurement, Evidence and Implications, Paris, OECD.

OECD, (2009), Implementation guidelines on evaluation and capacity building for local and micro regional level - "the Hungary case", OECD publication.

OECD (2005), 'Modernising Government: the way forward', OECD, Paris

OECD (2005). e-Government for better government. Paris, France: OECD.

OECD (2004) The Economic Impact of ICT: Measurement, Evidence and Implications, OECD, Paris.

OECD,(2003) e-Government Studies, 'The e-Government Imperative', www.oecd.org

Parra-Lopez, C., Groot, J.,C.,J., Carmona-Torres, C.& Rossing, W., A., H., (2009). "An integrated approach for ex ante evaluation of public policies for sustainable agriculture at landscape level". Land use Policy. 26(4), pp:1020-1030, Ed. Elsevier.

Perrin, B., (2006). Moving from Outputs to Outcomes: Practical Advice from Governments Around the World. Managing for performances and results series. Report sponsored by World Bank and the IBM Center for the Business of Government.

PITER, (2012). PITER - Piano Telematico della Regione Emilia Romagna, Programma Operativo 2012. Delibera della Giunta Regionale n°344/2012, del 26 Marzo 2012.

Rothig, P., (2004), WiBe 4.0: Recommendations on Economic Efficiency Assessments in the German Federal Administration, in Particular with Regard to the Use of Information Technology, KBSt Publication Series.

Rothig, P., (2010), The WiBe Framework: overview of an European measurement apprach to eGovernment Efficiency Assessment and User Satisfaction, KBSt Publication Series.

Saaty T.L., (2008), Decision making with analytical hierarchy process, International Journal Services Science, Vol.1, No. 1, Ed. Interscience Enterprise.

Saaty, T. L., & Vargas, L. G. (2012). A New Macroeconomic Forecasting and Policy Evaluation Method. Models, Methods, Concepts & Applications of the Analytic Hierarchy Process, 187-201.

Savoldelli, A., Codagnone, C., & Misuraca, G., (2012)., Explaining the eGovernment Paradox: An analysis of two decades of evidence from scientific literature and practice on barriers to eGovernment. Paper presented in: ICEGOV 2012, October, New York.

Savoldelli, A., Codagnone, C., & Misuraca, G., (2013), Assessing the impacts of ICT-enabled services in the public sector: a participatory evaluation process to overcome the eGovernment paradox. Paper presented at the EIBURS-TAIPIS, Urbino (IT), 18-19 April.

Schmoldt, D., L., Peterson, D.L., & Smith D., L., (2001). The analytic hierarchy process and participatory decision making. In "Decision Support - 2001", a collection of proceedings of the 17th Annual Geographic Information Seminar.

Stanirimovic, D., & Vintar, M., (2012), Evaluation of eGovernment Policies: Overlooked Aspects of Public Interest, Paper Presented in SOTICS 2012: The Second International Conference on Sociel Eco-Informatics.

Suggett, D., (2011), The implementation challenge: strategy is only as good as its execution, Occasional paper n°15, The Australian and New Zealand Schiool of Government.

Svensson, J., & Pettersson-Libdom, P., (2008), Impact Evaluation of Service Delivery Programs: Methods and Concepts for Impact Assessments in Basic Education, Health, Water and Sanitation. Paper prepared for the African Economic Research Consortium. Stockholm University.

Titah, R., & Barki, H., (2005). eGovernmnet Adoption and Acceptance: a Literature Review, Chaire de la Chaire du Canada en implementation et gestion des technologies des information. HEC Montreal. Available at http://neumann.hec.ca/igti/cahiers%20de%20recherche/chaireIGTIcahier0503.pdf .

Treasury Board of Canada (2004). Performance Measurement for the Government online Initiative. This document is available in alternative formats and on the Treasury Board of Canada Secretariat's Web site at the following address:www.tbs-sct.gc.ca

UN - DESA (United Nations Department of Economic and Social Affairs), (2010). eGovernmnet survey 2010: leveraging eGovernmnet at a time of financial and economic crisis.

Van Ryzin, G., (2009). Outcome, Process and citizens' trust of the civil services. Paper prepared for the 10th National Management Research Conference, October 1-4. Ohio State University, John Glenn School of Public Affairs, Columbus, USA.