Desenvolvimento de um Portal de Dados Abertos e de Uma Plataforma "Data Space" Para o Data Lake do Município da Maia

Abstract

A Câmara Municipal da Maia (CMM) possui um data lake com bastantes conjuntos de dados que abrangem uma grande variedade de informação, e está a pensar aproveitá-los através da criação de duas plataformas de partilha de dados.

Um portal de dados abertos com o objetivo de disponibilizar informação acessível a todos os cidadãos, seguindo standards europeus, e uma plataforma de espaço de dados para o projeto OMEGA-X, que inclui uma infraestrutura federada e uma marketplace de dados e serviços.

O desenvolvimento do portal de dados abertos estava em curso, e uma plataforma contendo alguns conjuntos de dados já tinha sido construido usando o OpenDataSoft. Porém, a metadata pertencente à camara municipal encontrava-se num nível básico, contendo apenas um modelo personalizado atendendo às necessidades do CMM, não cumprindo standards europeus. Adicionalmente, todo o trabalho desenvolvido pela câmara tinha sido feita diretamente pela interface da plataforma, o que, tendo em conta a dimensão do projeto, comprometia a sua sustentabilidade.

Relativamente ao sspaço de dados, a câmara municipal ainda estava numa fase preliminar, explorando o Sovity, o software indicado para o projeto OMEGA-X, e a compreender a sua complexidade.

Esta dissertação tem como objetivo ajudar o CMM em ambas as soluções de partilha de dados. No caso do portal aberto de dados, dar um passo importante na evolução dos metadados, evoluindo de um modelo específico à Maia para uma estrutura alinhada com os standards europeus e que maximiza as funcionalidades disponibilizadas pelo OpenDataSoft, assim como, desenvolver pipelines bem documentas para uma gestão automatizada da plataforma. Para o data space, pretende desenvolver uma prova de conceito usando o Sovity para estudar a sua complexidade.

Os resultados serão, posteriormente, testados. Para o portal de dados abertos, os princípios do FAIR vão ser usados para aferir a qualidade dos dados e metadados e as metricas definindas num estudo publicado pela universidade de Southampton para avaliar o estado do portal. Para avaliar a prova de conceito do data space, três casos de uso foram definidos.

Alternate abstract:

The Maia City Council (CMM) owns a data lake containing many datasets covering a wide range of information, and it is planning to leverage this data by creating two data-sharing platforms.

An open data portal with the goal of providing accessible information to all citizens, following European Standards and a Data Space platform for the OMEGA-X project, including a federated infrastructure and a data and service marketplace.

The development of the open data portal was underway, and a platform containing some datasets had already been built using OpenDataSoft. However, the metadata owned by the city council was still at an elementary level, containing only a custom model to attend to the CMM’s needs, not complying with European standards. Additionally, all the work carried out by the city council had been done directly on the platform’s interface, which, considering the dimensions of the project, compromised sustainability.

For the data space, the city council was still doing preliminary work, exploring Sovity, the software indicated for the OMEGA-X project, and understanding its complexity.

This dissertation aims to assist the CMM in both of these data-sharing solutions. For the open data portal, it seeks to take an important step in the evolution of the metadata, evolving from a single Maia-specific model to a structure aligned with European standards and that maximises the functionalities provided by OpenDataSoft, as well as to develop well-documented pipelines for an automated management of the platform. For the Data Space, it intends to develop a proof of concept using Sovity to study its complexity.

The results will then be tested. For the open data portal, the FAIR principles will be used to assess the data and metadata quality, and the metrics defined in a study published by the University of Southampton to evaluate the state of the portal. To evaluate the data space proof of concept, three use cases were defined.

Abstract (AI English translation)

Information popover about translation disclaimer

Maia Municipal Council (CMM) has a data lake with many data sets that cover a wide variety of information, and is thinking about taking advantage of them by creating two data sharing platforms.

An open data portal with the aim of making accessible information available to all citizens, following European standards, and a data space platform for the OMEGA-X project, which includes a federated infrastructure and a data and services marketplace.

Development of the open data portal was ongoing, and a platform containing some datasets had already been built using OpenDataSoft. However, the metadata belonging to the municipal council was at a basic level, containing only a personalized model meeting the needs of the CMM, not complying with European standards. Additionally, all the work carried out by the chamber had been done directly through the platform's interface, which, taking into account the size of the project, compromised its sustainability.

Regarding the data space, the city council was still in a preliminary phase, exploring Sovity, the software recommended for the OMEGA-X project, and understanding its complexity.

This dissertation aims to help CMM in both data sharing solutions. In the case of the open data portal, taking an important step in the evolution of metadata, evolving from a model specific to Maia to a structure aligned with European standards and which maximizes the functionalities provided by OpenDataSoft, as well as developing well-documented pipelines for automated management of the platform. For the data space, we intend to develop a proof of concept using Sovity to study its complexity.

The results will later be tested. For the open data portal, FAIR principles will be used to assess the quality of data and metadata and the metrics defined in a study published by the University of Southampton to assess the status of the portal. To evaluate the data space proof of concept, three use cases were defined.