Open data in Service design

Abstract

This paper proposes a review of the current approaches to encouraging the creation of services based on data, and an analysis of the creation of services from two open data platforms, in the UK and in Singapore. It also explores the roles that the (open) data can have in the design of services based on a theoretical framework of service innovation. Data can play different roles: (i) the service is based on data, (2) the service uses data as a resource, and (3) the service is validated or enriched with data but the data is not directly used or is not directly visible in the service. This offers new opportunities for the reuse of data and suggests a different approach to measuring the impact of opening datasets beyond the mere number of services created.

Headnote

Abstract: This paper proposes a review of the current approaches to encouraging the creation of services based on data, and an analysis of the creation of services from two open data platforms, in the UK and in Singapore. It also explores the roles that the (open) data can have in the design of services based on a theoretical framework of service innovation. Data can play different roles: (i) the service is based on data, (2) the service uses data as a resource, and (3) the service is validated or enriched with data but the data is not directly used or is not directly visible in the service. This offers new opportunities for the reuse of data and suggests a different approach to measuring the impact of opening datasets beyond the mere number of services created.

Keywords: open data, service design, data-driven service, innovation

1 Introduction

The development of a data driven economy has been a major orientation of economic policies over the past few years based on (i) the wider availability of data promoted in particular by the Open Data movement (Chui et al., 2013) and (ii) the development of dedicated tools to support heterogeneous data and data in large quantities (Big data). Reports anticipate the creation of enormous amounts of economic activity and growth opportunities (e.g. Buchholtz et al., 2014). However the promise of the data-driven economy lies to a large extent in the development of new services (Deloitte, 2011). The return on investment of open data policies for instance should be evaluated from the services created based on open data sets. Open data promoters couple more and more open data initiatives with actions dedicated to the promotion of the datasets for the creation of new services (Foulonneau et al., 2014). Nevertheless the results in terms of services created remain below the expectations of open data promoters. Indeed most services created are not sustainable and / or do not use the variety of datasets. They are to a wide extent relying on a limited number of very popular datasets. In order to make the promise of the data-driven economy a reality, it is therefore necessary to increase reuse and value extracted by services from data. Our hypothesis is that service innovation approaches can help understand the mechanisms that drive the creation of services. We therefore propose to (i) review the current approaches to encouraging the creation of services based on data, to (ii) examine the creation of services from two open data platforms, in the UK and in Singapore, finally to (iii) analyse the roles that the data can have in the design of services based on a theoretical framework of service innovation.

2 Approaches to encourage the creation of services based on open data

Existing open data initiatives are often accompanied to some extent of actions dedicated to facilitating the creation of services.

2.1 Opening datasets:

- Identification of datasets to open: To select the datasets to open, it is either possible to investigate which datasets are already available, whose IPR (Intellectual Property Rights) conditions would allow a publication under an open licence, and whose quality is sufficient. It is also possible to investigate what citizens and potential reusers are interested in (EC, 2013).

- Data accessibility: In order to make datasets accessible, Open Data portals at regional, national and international level aggregate metadata on datasets.

- Datasets discoverability: The harmonization of metadata vocabularies (Martin et al., 2013c) used to describe datasets could help the discovery of datasets.

- Intellectual Property Rights: To lower the barrier related to intellectual property rights and reuse conditions, some platforms (like the Singapore open data platform (http://data.gov.sg)) insist on having well documented IPR on each dataset. Other platforms impose very open licences (the Europeana initiative (www.europeana.eu) has set CC0 license which all data providers have to sign in order to have their data included in the platform) to avoid to the reusers to address heterogeneous reuse conditions when reusing multiple datasets in conjunction (e.g. UK Open Government Licence, http://goo.gl/YZfzI4).

2.2 Publicizing datasets and applications

- Publicizing new datasets: When data producers publish new datasets, they have to make the datasets known to potential reusers to optimize the chances that reusers will take advantage of datasets and to facilitate the creation of new services. They advertise on virtual community channels the information and reach potential reusers for their datasets.

- Publicizing services: In order to publicize data reuse and encourage service usage, most open data portals include a section to show services and applications that use the datasets, like for example the App showroom on the Berlin Open Data portal "Anwendungen" (http://goo.gl/p6DE0), the "Apps" section (http://data.gov.uk/apps) on the data.gov.uk portal, the "Mobile Apps Gallery" (http://apps.usa.gov/) section on the data.gov portal, the "Application Showcase" on the Singapore portal (http://goo.gl/pN27Bh), or the "Application showcase" section (http://goo.gl/bNXFw) of the World Bank data portal illustrate the systematic addition of a section dedicated to services.

2.3 Training & support

- Training: The release of open data is not sufficient to make them accessible and understandable by reusers. In order to lower the technical barrier as well as the time required to develop a new service, certain data publishers create APIs (Cohen, 2013). Reusers therefore do not have to address format heterogeneity and the learning curve to reuse open datasets is shorter. Over time, the number of available training opportunities for open data users and reusers is continuously growing. For example, the World Bank provides various training services to help potential reusers how to get data and to represent data it into maps (http://goo.gl/dSzplE). The Open Knowledge Foundation offers courses on open data (http://okfn.org/training/), for stakeholders who may not have extensive technical knowledge but wish to work on the data (e.g. data journalists). The European commission publishes an important set of online training material focusing on the linked open data on the JoinUp platform (http://training.opendatasupport.eu).

- Support: In its report for the German Open Data platform (Klessmann et al., 2012), the Fraunhofer FOKUS institute advises the creation of a platform to support reusers and help them reuse the data. The US "Project Open Data" (http://project-open-data.github.io ) provides public administration and data-reusers with a large set of documentation and tools to assist them in data opening and reuse.

2.4 Competitions

- Hackathons and off-site competitions : Hackathons (i.e. "Open Public Data Hackathon" organised in April 2014 by the Greek government ) and off-site competitions are among the initiatives organised to show the variety of services that can be created based on open datasets. They are effective at raising the awareness of developers and service creators on the potential for reuse of the open datasets.

- Calls for contribution: In order to ensure the creation of sustainable services however the Singapore government organizes the co-funding of services (Calvin, 2013). All the same, the European Commission co-funds the development of services to take advantage of geographic information coming from the Galileo satellite (http://goo.gl/VGhZM7) and services that reuse open datasets (European Commission, 2013).

The deployment of all these strategies has enabled the creation of many services based on open datasets. In many cases, services are delivered through applications. We nevertheless use the generic term of services as "a set of deeds and acts performed by or on behalf of an agent for the benefit of a citizen, a business or another agent" as an extension of the definition provided in the Core Public Service Vocabulary ISA (2013).

In the next section, we analyse the reuse of open datasets by services.

3 Analysis of the creation of services

We observed the open datasets and the services created based on open data listed on the UK Open Data portal as well as the Singapore Open Data portal. These different cases were selected due to (i) the expressed willing of the supporting authorities to promote the creation of services based on the open data sets, and due to (ii) the availability of data concerning open datasets and developed services. In September 2014, the data.gov.uk portal lists 349 apps and the Singapore open data portal around 138.

On the UK portal, the domains of the applications (Figure 1) include economy (e.g., Property prize , jobmarket), environment (e.g., Use of renewable energies), transparency (e.g., Availability of governmental financial data with the famous application « Where does my money go? »), society (Criminality map), local services (e.g., best surgery services next to me, postal code where I am, fuel station search), education (e.g., geological layers where I stand), and citizen life (e.g., elections results, « UK arms export licence browser »).

Within the investigated applications, around 78% only reuse a single dataset. Sketchmap.co.uk uses 19 datasets. The most reused datasets are Code-Point® Open and the National Public Transport Access Nodes (NaPTAN). Also, it should be noted that transportation datasets are indeed popular.

On the Singaporean portal, available applications concern mainly navigation and mobility (traffic, maps), Tourism, Environment, Security (Emergency contacts points), Religion (places of worship), Business (Multiple listing of companies registered in Singapore) and open jobs in public administrations.

138 applications, all of them free, are listed. Some of them available on multiple OS (iOS, Android, etc.). Only 13 apps are using more than one dataset and 89 (85%) of them use only one dataset. The most re-used data set is "Singapore map (OneMap) data" from Singapore Land Authority; used in 33 apps. Datasets related to Traffic and parking data are the second used in 19 apps.

Figure 1 illustrates the domain of the applications created based on datasets from the data.gov.uk and data.gov.sg portals. Figure 2 shows the thematic distribution of the datasets available on data.gov.uk and data.gov.sg portal.

Although categories are not exactly similar, the comparison in the both cases of UK and Singapore of both figures shows the discrepancy between the low number of geographic and transportation datasets and the very large proportion of transportation services. While transportation datasets appear to have a high reusability potential, education datasets attract less reusers in proportion of their quantity.

In Singapore, even if there are more than more than 5000 publicly-available datasets from 50 public agencies, no more than 100 of them are used by the around 100 apps. In UK, we can make the similar observation about the 17023 datasets and the 349 apps.

Although there is no direct equivalence between the domain of the dataset and the domain of the service that reuses it, this suggests the quite uneven potential of datasets for reuse.

The domain is clearly not the only element to take into consideration when assessing the reusability of a dataset.

Therefore the way in which we propose to increase data reuse and the number of services created is by understanding better the service creation process and the way in which it is possible to integrate data in this process. Open data portals describe many datasets, available in a variety of formats and with many different access modes (Martin et al., 2013c). There are various data collection mechanisms for reusing datasets from the Web, including the open Archives Initiative Protocol for Metadata Harvesting (http://www.openarchives.org/pmh/) used for instance in the Europeana digital library, Linked Open Data (Heath et al., 2011), data dumps, and data wrappers. However, whichever the type of data and the data access mechanism, a number of similar issues are raised, in particular related to data reusability (e.g., data quality). Data producers make their data available for reuse but reuse does not always happen. It is therefore necessary to understand the service creation process and how data sources can be integrated in this process.

4 The service design process

The service design process is composed of multiple steps (Vidou, 2013), i.e., idea generation, maturation, and concept evaluation (Figure 3). It should take into consideration different aspects, including the context, the target, the synopsis, the innovativeness and sustainability, the resources, and the service system. Before it becomes an innovation, an idea has to gain its spurs: Does it have a market, technological or research potential? Is it worth investing in its development? The lifecycle of an idea is composed of 3 phases of their life: (i) birth, (ii) maturation and (iii) evaluation. This is an iterative process. Jetzek et al. (Jetzek et al., 2013) also present the innovation process around three main phases: idea generation, idea conversion and idea diffusion. The idea conversion phase corresponds to the maturation phase in which the idea is transformed into new product, service, business processes or behavioural innovations. The diffusion phase coincides on the evaluation phase as it deals with the spreading of the innovative product within and outside the company.

The (i) ideation phase corresponds to the birth of the idea. An idea can be born spontaneously or from the systematic exploration of various fields of innovation. It can be triggered by a stimulus at the occasion of a call for ideas or an ideation contest.

The (ii) maturation phase consists in exploring the idea related issues, validate its options or not, make some investigation which other technologies and services are out there already. The idea maturation can be individual - the leader explores his idea in his personal space, with dedicated tools; shared - the leader shares his idea with specific persons, requesting specific contributions; or open - the idea is then available to all (at a department, enterprise or cluster level).

When the idea has reached a level of maturity satisfying its leader, the potential of the idea can be assessed by a group of experts who can decide to invest in its development - this is the (iii) evaluation phase.

During the idea lifecycle, some aspects of the idea should be addressed in order to avoid missing elements (Figure 4): synopsis, context, target, resources, service system, innovativeness and sustainability. The Synopsis is the minimum level of description of the service, the summary of the concept. The context dimension describes the context in which the service is delivered (time, space technological components, regulatory context: norms, standards...).

The target describes the customers of the service and the reason why they would buy it.

The resource component describes the type of resources required to deliver the service (human resources: skills and competences, technological resources, process & organizational resources, norms and standards, partners, financial resources...).

The service system component describes the way in which resources are combined to deliver the service to the target in the context (key activities, key partners as stakeholders)

The innovativeness and sustainability highlights the innovative aspects of the service system through its ingredients and the expected economical, societal & environmental impacts.

For each dimension, elements can be at different levels of maturity to support the progressive maturation of the service (basics: simple description based on assumptions or insights; argued: detailed and comprehensive description, checked: detailed description, thought through, checked against literature or facts).

In innovation, there are multiple opportunities for the creation of innovative services. A demographic change, such as the increase of the elderly population can lead to opportunities for the creation of services for retired people. A technology such as indoor GPS can lead to the creation of services based on the new possibilities offered by the availability of that technology. Just like a technology and a demographic change, the availability of data can create opportunities for the creation of new services.

5 The role of the dataset in the services

As summarized in Figure 5, data can therefore play different roles in the service design process: 1) The service is based on data, 2) the service uses data as a resource, and 3) The service is validated or enriched with data but the data is not directly used or is not directly visible in the service.

5.1 Service based on data

When the data is used as impulse to the service ideation process, it represents the core of the service concept. The objective of the ideation process is to determine with one or more datasets which services could be designed based on them. The data producer should wonder who currently uses the data; who else could be interested by these data; and if there is any combination of these data with other data which could be of interest to a stakeholder.

The service can for instance allow visualizing the data. The Website publicspending.net for instance allows citizens to view budgetary data. The nosdeputes.net service shows the activity of French members of parliaments. The objective of these services is to provide a new didactic access point to the data and thus enable transparency. The handimap.org service in the French cities Rennes and Montpellier show paths through the cities for disabled citizens. By combining public and private data sources on the creation of companies in Europe, the European SPOCS project has shed a new light on the comparative ease of creating a new business in European countries (Foulonneau et al., 2013a). The Narrative science company generates texts from data so as to make it more user friendly (http://goo.gl/Cmwzf2). It is for instance possible to generate sport news articles from the raw results of local competitions which would not be covered by newspapers.

Finally, it is possible to give a new meaning to the data. Google ngrams (https://books.google.com/ngrams) for instance benefits from the Google book digitization program. In this context, Google has digitized millions of books from various countries at various times. It then applies an Optical Character Recognition process so as to enable full text search functionalities in the books. By combining the bibliographic data (including the date of publication) and the individual words used in each book, Google ngrams allows visualizing the evolution of the use of particular words over time (Figure 6).

At the maturation phase, it is then necessary to analyse the characteristics of the datasets and their impact on the feasibility of the service. This includes the update frequency, the data quality, such as its reliability, completeness etc. (see Stvilia, 2007), the data source, its maintenance processes, intellectual property rights attached to the data and its conditions of use, its cost, its accessibility, including its technical accessibility (e.g., API, data dump ...) as well as its formats (e.g., RDF/XML, JSON, spread sheet), its interoperability with other datasets, typically to mix it with third party datasets, and its documentation including the documentation of its underlying semantic model in order to adequately interpret and use it.

Indeed the update frequency and the maintenance processes attached to the data may be critical and impact the quality of the service. This can jeopardize its viability. In addition, specific processes may have to be in place in order to ensure the collection of updated data for instance. The cost has a direct impact on the business model of the service, while the lack of interoperability with other datasets of interest may lead to additional costs to make aggregate them in a single service.

In this type of services, the data represents the main resource or one of the main resources. Its characteristics should therefore be analysed in detail at the occasion of the maturation phase of the service design.

5.2 Services with data as resources

In other types of services, the concept is defined at the ideation phase without any specific relation with the datasets. However when investigating the feasibility of the service at the maturation phase, the datasets have to be taken into consideration as necessary resources. Location and traffic data for instance is not the core of a delivery service concept. However they are resources that will help design the service.

Data enrichment can use for instance Wikipedia for the translation of concepts or a dataset of postcodes to automatically fill the city in an address form. In a service designed to assess the reading difficulty of educational texts (Foulonneau et al., 2013b) we used a lexical base (WordNet) to identify the relation between words in contiguous paragraphs so as to define breaks in the texts. The final system does not display the external data. It may even work without the external dataset (with simple sentence similarity metrics based on exact word matching for instance). However the external dataset increases the quality of the final system.

As for the services based on data, the datasets should be analysed in terms of intellectual property rights, reuse conditions, maintenance processes, update frequency, technical accessibility, formats, data quality, costs, interoperability with other datasets, and documentation in particular.

A major challenge for open datasets for instance is that in many cases the reuse conditions of the datasets are not clear (Martin et al., 2013a). When a service has to reuse multiple datasets, the multiplicity of reuse conditions may increase the barrier for the reusers. The Europeana digital library has therefore promoted the CC0 licence to ensure that all datasets would be reusable without limitation.

5.3 Service validation

Finally datasets may be used in the design phase of the service but not in the service itself. Indeed, in the concept validation phase it is possible to use external datasets only for testing a service concept or validating data which are already hold by the service designer.

Datasets can be used for validating the data already used in the service. A dataset of postcodes can be used to validate postcodes provided by users in a form. Recommendation systems are often tested against standard datasets, such as MovieLens (http://grouplens.org/datasets/movielens/) which has gathered ratings of users over many movies. Authors of new algorithms can then test their algorithm against the dataset to verify that it can accurately predict the ratings provided by users and compare its performance to the performance of the numerous other algorithms which have been tested against the MovieLens dataset and for which performance has been reported in scientific publications.

Datasets can also be used to validate a business model, through gathering economic indicators, typically from statistical institutes that publish their datasets. Simulation environments which require many datasets to recreate the context of execution of a service are often used to predict the difficulties or identify the optimal distribution of resources, for instance in traffic related services.

In these cases the external datasets do not appear in the final service. However they play a critical role to increase the quality of the service and ensure its viability.

6 Conclusion

This paper proposed a review of the current approaches to encouraging the creation of services based on data, an analysis of the creation of services from two open data platforms, in the UK and in Singapore. Also, it explored the roles that the data can have in the design of services based on a theoretical framework of service innovation. A market has emerged with companies dedicated to supporting access and the design of new services based on data (e.g., Enigma.io, Narrative Science). Even Google offers a visualization service for open data sets (http://www.google.com/publicdata/). More and more data access portal build APIs and mechanisms that lower the barrier for service designers. Dedicated software has been designed such as the Callimachus platform to reuse Linked open Data (http://callimachusproject.org). In addition, many platforms also organize competitions, such as Hackathons (http://goo.gl/cINX7L), display successful services or even co-finance the development of services based on the data (e.g., in Singapore (Calvin, 2013)). Competitions especially often focus on the first type of service for which data are the core resource. However it is important to understand the roles that the data can have in a service and most importantly that they can also help at the maturation and the validation phases of the service design. This offers new opportunities for the reuse of data and suggests a different approach to measuring the impact of opening datasets beyond the mere number of services created. Indeed in the future it would be necessary to ensure that the benefit of opening data can be adequately measured and that datasets reuse is not only a matter of promotion but also a matter of asset value, i.e., the characteristics of the datasets are critical to ensure their effective reuse by service designers.

References

References

Buchholtz S., Bukowski M., Sniegocki A., (2014). "Big & Open Data in Europe: A growth engine or a missed opportunity?" demosEUROPA & WISE Institute, ISBN: 978-83-925542-1-9, Jan 2014.

Calvin M.L. Chan (2013). "From Open Data to Open Innovation Strategies: Creating e-Services Using Open Government Data", 46th Hawaii International Conference on System Sciences.

Chui M., Farrell D., Van Kuiken S. (2013). "Generating Economic Value through Open Data". Beyond Transparency: Open Data and the Future of Civic Innovation. October 2013. ISBN-10: 0615889085.

Cohen R. (2013). "Obama Signs Open Data Executive Order: U.S. Government Data To Be Made Freely Available". Forbes, May 2013. Available at http://goo.gl/sb7cSr

Deloitte Analytics Institute (2011). "Unlocking growth: How open data creates new opportunities for the UK".

European Commission (2013). "Competitiveness and innovation framework programme (CIP)". ICT policy support programme, ICT-PSP work programme 2013.

European Commission (2013). "Results of an online consultation on the guidelines on recommended standard licences, datasets and charging for the re-use of public sector information". Final report http://goo.gl/LeQ7jg

Foulonneau M. et al. (2013). "SPOCS: A semantic interoperability layer to support the implementation of the service directive". Proc.of ECEG'2013, 13th European Conference on e-Government. June 2013, Como, Italy.

Foulonneau M., Martin S., Turki S. (2014). "How Open Data Are Turned into Services?", Proc. Of IESS 2014: 31-39, Exploring Services Science. Geneva, Switzerland, February 2014. Springer 2014 Lecture Notes in Business Information Processing ISBN 978-3-319-04809-3.

Foulonneau M., Ras E., Abou Zeid E., Atéchian T. (2013). "Reusing Textual Resources in Educational Assessment: Adding Text Readability Metrics to Learning Metadata". Dublin Core conference 2013, Lisbon.

Heath T., Bizer Ch (2011). "Linked data: Evolving the web into a global data space". Synthesis lectures on the semantic web: theory and technology 1.1 (2011): 1-136.

ISA (2013) Core Public Service Vocabulary 1.01, designed under the auspices of the ISA programme of the European Commission (Interoperability Solution for European Public Administrations), available at http://goo.gl/cThwGX

Klessmann et al., (2012). "Open Government Data Germany", Short Version of the Study on Open Government in Germany Commissioned by the Federal Ministry of the Interior. July 2012.

Martin et al., (2013), "Risk Analysis to Overcome Barriers to Open Data". Electronic Journal of e-Government Volume 11 Issue 1 2013, (pp348 -359), available at www.ejeg.com

Martin et al., (2013). "Open Data: Barriers, Risks, and Opportunities". Proc. Of ECEG'2013, 13th European Conference on eGovernment, Como, Italy, June 2013. ISBN: 978-1-909507-22-7.

Martin, S., Foulonneau, M., Turki, S. (2013). "1-5 stars: Metadata on the Openness Level of Open Data Sets in Europe". In Metadata and Semantics Research 2013. Communications in Computer and Information Science, Springer Verlag, Heidelberg.

Stvilia et al., (2007). "A framework for information quality assessment". Journal of the American Society for Information Science and Technology 58.12 (2007): 1720-1733.

Vidou, G. (2013). "The Service Value Pathway: the 3-6-3 tool", The XXIV ISPIM Conference, Helsinki, 2013.

Buchholtz S., Bukowski M., Sniegocki A., (2014). "Big & Open Data in Europe: A growth engine or a missed opportunity?" demosEUROPA & WISE Institute, ISBN: 978-83-925542-1-9, Jan 2014.

Calvin M.L. Chan (2013). "From Open Data to Open Innovation Strategies: Creating e-Services Using Open Government Data", 46th Hawaii International Conference on System Sciences.

Chui M., Farrell D., Van Kuiken S. (2013). "Generating Economic Value through Open Data". Beyond Transparency: Open Data and the Future of Civic Innovation. October 2013. ISBN-10: 0615889085.

Cohen R. (2013). "Obama Signs Open Data Executive Order: U.S. Government Data To Be Made Freely Available". Forbes, May 2013. Available at http://goo.gl/sb7cSr

Deloitte Analytics Institute (2011). "Unlocking growth: How open data creates new opportunities for the UK".

European Commission (2013). "Competitiveness and innovation framework programme (CIP)". ICT policy support programme, ICT-PSP work programme 2013.

European Commission (2013). "Results of an online consultation on the guidelines on recommended standard licences, datasets and charging for the re-use of public sector information". Final report http://ec.europa.eu/digitalagenda/en/news/results-online-survey-recommended-standard-licensing-datasets-and-charging-re-use-public-sector

Foulonneau M. et al. (2013). "SPOCS: A semantic interoperability layer to support the implementation of the service directive". Proc.of ECEG'2013, 13th European Conference on e-Government. June 2013, Como, Italy.

Foulonneau M., Martin S., Turki S. (2014). "How Open Data Are Turned into Services?", Proc. Of IESS 2014: 31-39, Exploring Services Science. Geneva, Switzerland, February 2014. Springer 2014 Lecture Notes in Business Information Processing ISBN 978-3-319-04809-3.

Foulonneau M., Ras E., Abou Zeid E., Atéchian T. (2013). "Reusing Textual Resources in Educational Assessment: Adding Text Readability Metrics to Learning Metadata". Dublin Core conference 2013, Lisbon.

Foulonneau, M., Turki, S., Vidou, G., Martin, S. (2013) "From Open Data to Data-Driven Services", ECEG'2014: 14th European Conference on e-Government. June 2014.

Heath T., Bizer Ch (2011). "Linked data: Evolving the web into a global data space". Synthesis lectures on the semantic web: theory and technology 1.1 (2011): 1-136.

ISA (2013) Core Public Service Vocabulary 1.01, designed under the auspices of the ISA programme of the European Commission (Interoperability Solution for European Public Administrations), available at http://goo.gl/cThwGX

Jatzek et al. (2013) "Data-Driven Innovation through Open Government Data". Journal of Theoretical and Applied Electronic Commerce Research. ISSN 0718-1876

Klessmann et al., (2012). "Open Government Data Germany", Short Version of the Study on Open Government in Germany Commissioned by the Federal Ministry of the Interior. July 2012.

Martin et al., (2013), "Risk Analysis to Overcome Barriers to Open Data". Electronic Journal of e-Government Volume 11 Issue 1 2013, (pp348 -359), available at www.ejeg.com

Martin et al., (2013). "Open Data: Barriers, Risks, and Opportunities". Proc. Of ECEG'2013, 13th European Conference on eGovernment, Como, Italy, June 2013. ISBN: 978-1-909507-22-7.

Martin, S., Foulonneau, M., Turki, S. (2013). "1-5 stars: Metadata on the Openness Level of Open Data Sets in Europe". In Metadata and Semantics Research 2013. Communications in Computer and Information Science, Springer Verlag, Heidelberg.

Stvilia et al., (2007). "A framework for information quality assessment". Journal of the American Society for Information Science and Technology 58.12 (2007): 1720-1733.

Vidou, G. (2013). "The Service Value Pathway: the 3-6-3 tool", The XXIV ISPIM Conference, Helsinki, 2013.