What Do Users Really Need? Participatory Development of Decision Support Tools for Environmental Management Based on Outcomes

Abstract

There is increasing demand from stakeholders for tools to support outcomes-based approaches in environmental management. For such tools to be useful, understanding user requirements is key. In Scotland, UK, stakeholders were engaged in the development of an Environmental Decision Support System (EDSS) to support the management of land and freshwater resources for multiple policy outcomes. A structured participatory engagement process was employed to determine stakeholder requirements, establish development principles to fulfil these requirements and road-test prototypes. The specification that emerged from this bottom-up process was for an EDSS to be spatially-explicit, free at the point of use, and mobile device compatible. This application, which is under development, does not closely resemble most existing published EDSS. We suggest that there is a mismatch between the way scientists typically conceptualise EDSS and the kinds of applications that are likely to be useful to decision-makers on the ground. Interactive mobile and web-based geospatial information services have become ubiquitous in our daily lives, but their importance is not reflected in the literature on EDSS. The current focus in environmental management on adaptive, stakeholder-centred strategies based on outcomes offers an opportunity to make better use of these new technologies to aid decision-making processes.

1. Introduction

Research in environmental science is often undertaken under the premise that scientific information and knowledge is necessary to inform environmental policy and management. This has led to the widespread development of computerized tools to bridge the divide between scientific analysis of the state of the environment (e.g., water quality, biodiversity, land use change) and environmental policy objectives (e.g., European directives on freshwater quality and terrestrial biodiversity conservation measures). These tools fall approximately into two types, though in practice there is a great deal of overlap. Decision Support Systems (DSS) are typically targeted at supporting policy implementation in a specific context, while Policy Support Systems or (PoSS) have broader aims, including policy formulation and strategy [1]. In the context of environmental management, we follow [2], and refer to both of these two types as Environmental Decision Support Systems (EDSS).

In theory, the relationship between science and policy is close and direct [3]. In practice, this is not always the case, especially when science identifies wicked problems [4] to which policy makers or environmental managers are unable to respond in conventional normative ways. Consensus is beginning to emerge around the need for integrative, adaptive approaches to environmental management as a means to tackle these kinds of intractable problems [5,6]. Adaptive environmental management seeks to integrate project design, management, and monitoring, to provide a framework to systematically test assumptions, promote learning, and supply timely information for management decisions (e.g., [7]). Involvement of stakeholders is essential to facilitate the processes of generating and sharing different knowledges to improve understanding of the effectiveness of management actions. Policy makers and land managers require better access to the results of scientific analysis, while scientific stakeholders need to better understand other stakeholders’ needs in order to structure and focus their research (e.g., [8]). Private sector stakeholders, like land-based businesses, are increasingly interested in easy to use web and mobile-based dashboard “business intelligence” tools to better manage their holdings, creating new demands and opportunities to bridge the relationship between policy and management, and the supply and provision of scientific analysis. A need is therefore emerging for new software tools and applications to respond to these demands, bringing new ways of working [9]. For example, up-to-date information on the state of the physical environment (moisture, erosion, crop growth), and land-based policies and incentives can be accessed directly by land managers through smartphone and tablet applications. This helps them play a more proactive role in environmental management and may reduce the need for intervention by regulators. At the same time, scientists can move from passive provision of information for policy makers, to on-the-ground facilitation of knowledge exchange between all stakeholders. Making applications web-based facilitates access, which might be expected to lead to faster and more widespread adoption by taking advantage of existing internet infrastructure and appealing to users of modern mobile devices.

In this paper, we discuss our recent progress in responding to the challenge of providing web-based digital tools that meet user needs for their adaptive management of natural resources. To this end, we address the following three key research questions:

1. How can we develop decision-support tools that align better with adaptive management and outcomes-based approaches to environmental management: what are the key requirements of such a system?

2. How can we make better use of well-established web-based and mobile devices and software for supporting environmental decisions?

3. How can we involve key stakeholders like scientists, regulators and land managers to better understand these requirements?

The paper is structured as follows. In the next section, we describe some of the most important limitations of EDSS as currently conceived from an adaptive management perspective. Subsequently, we show how we have tried to address these limitations and answer our research questions by engaging policy stakeholders, regulators and resource managers to co-develop an environmental decision-support application for understanding the effectiveness of environmental policy interventions in river catchments in Scotland. We present the results of this process and finish with a general discussion summarizing key points and lessons learnt. Finally we offer some brief recommendations for future development of EDSS.

2. Background

The application of computerized tools to these kinds of problems has a long history (see e.g., [10,11,12]). The use of computer modelling to facilitate adaptive environmental management was advocated by [13]. Early DSS (see e.g., [14,15]) were conceived as computerized tools to manage operational decisions where either the decision formulation or the solution, or both, were arguable or directly contested [2]. The literature contains many different types of EDSS, depending on the kinds of decisions they are intended to support, their anticipated or declared end user, and the stage of the environmental policy process to which they are directed. Some, like the SimLucia model ([16,17]), developed on behalf of the United Nations Environment Programme (UNEP), are intended to allow high-level policy makers to formulate appropriate long-term responses to environmental change. Others, like QUICKScan ([18]) are targeted at more local scales and over shorter timeframes e.g., to help multiple stakeholders to negotiate, compare options and understand trade-offs in the implementation of concrete policies. Others, such as the COLLAGE tool, are more specific. COLLAGE is designed to help local stakeholders plan renewable energy installations by balancing generation capacity against local spatial planning concerns ([19]). While the involvement of end-users is clearly a key aspect of many such systems, this does not necessarily imply the democratization of knowledge and decision-making that lies at the heart of stakeholder-centred approaches to environmental management, i.e., in participatory, collaborative or mediated modelling (e.g., [20,21,22]). An EDSS designed under the conventional paradigm of DSS is a tool to help the competent authorities solve environmental problems, but does not necessarily emphasize sharing and co-construction of knowledge with stakeholders or social learning (sensu [23]) as a strategy for managing disagreement resulting from the diverse perspectives and requirements of multiple stakeholders. This means that frequently, such systems are not targeted at stakeholder needs [24]. From an adaptive management point of view, in which it is often desirable or even essential to share knowledge effectively amongst diverse stakeholder groups, this is problematic. Recently, digital catchment observatories have been suggested as a means to improve knowledge sharing and co-construction with stakeholders [25].

At the same time, many EDSS, ostensibly intended for non-scientific stakeholders to pick up and apply to their specific problems or needs, are not used for that purpose [2,26,27]. While the reasons for the lack of uptake by intended end users are diverse [2,28] lack of stakeholder involvement at the design stage is clearly a significant factor. Volk et al [24], in their analysis of four EDSS in landscape and catchment management contexts, concluded that “the appropriate and methodological stakeholder interaction and the definition of ‘what end-users really need and want’ have been documented as general shortcomings of all four examples of DSS.” Understanding and managing the diverse requirements a software application needs to meet—the requirements problem—is recognized as a major challenge in software development generally [29]. Responding to this challenge by improving the integration of the end user in the development process is a key underlying motivation of the User-Centered Systems Design (UCSD) paradigm [30,31]. However, while user-centered approaches have become mainstream in software development circles, they still lack general application in an EDSS context.

Further, since publication of [2], the role of software in supporting decisions of various kinds in everyday life has grown substantially. Mobile and web-based applications have become ubiquitous in all kinds of contexts, e.g., purchasing or contracting goods and services, banking, navigation, social networking etc. Typically these tools are mobile, web-based, built on Free and Open Source Software (FOSS), and touch user interface-optimized. Yet EDSS for the most part, lag behind the innovation curve. Most EDSS are still stand-alone desktop systems, many require expensive proprietary software and frequently are not optimized or not available for touch-enabled devices. In agriculture, a new generation of decision-support tools has begun to emerge, in the form of Farm Management Information Systems [32,33,34], allowing farm managers to adapt their operations to variables like temperature and precipitation, market prices or policy measures. However, there remains a substantial mismatch between the computationally intensive and conceptually challenging modelling approaches typically used by the scientific community (e.g., [1,35]), and the lightweight, web and mobile-based, touch-enabled applications for smart devices that we all use in our daily lives.

Finally, a key limitation of conventional EDSS from our point of view relates to the difficulty of applying them in the context of outcomes-based approaches i.e., the environmental, social or financial improvements that management actions aim to make. The likely result, in terms of environmental improvement or otherwise of a particular measure, should be a determining factor in the choice of measure and where implemented. This requires us to understand not only the potential spatial distribution of particular environmental variables in the landscape e.g., water body or terrestrial habitat status as determined by environmental policies, in order to suggest locations for management measures and interventions, but also the causal logic that connects these measures to the desired outcomes. Conventional EDSS approaches have emphasized the former, while approaches like logic modelling (also known as results chain logic modelling) are frequently applied to the latter [36,37]. Logic modelling, which has its origins in program theory [38] is widely used for planning and evaluation of environmental and agricultural policy measures in the UK and elsewhere [39]. At present, however, we are unaware of any EDSS software or framework that successfully integrates the aspatial process-based approaches found in logic modelling software like Miradi with the spatial, Geographical Information Systems (GIS)-type approach found in EDSS for rural planning at the level of the management unit (e.g., [10]).

In the following paper, we present our recent work in this area, and argue that a reappraisal of the process of developing software for EDSS is necessary to take into account adaptive management and outcomes-based approaches to environmental decision-making and the wide range of new smart applications.

3. Case Study Background

The work presented here forms part of a long-term Scottish Government Strategic Research Programme (SRP) project which aims to understand and improve the management of Scotland’s land and freshwater resources for multiple policy and management outcomes. In Scotland, the Land Use Strategy aims to take a more integrated approach to management of natural capital to deliver multiple benefits [40]. Since 2007, Scottish Government policy objectives have been framed as a set of national level outcomes to guide all policy, management and applied research [41]. This emphasis on outcomes stems from the increasing interest across research, policy and management communities to improve not only how we plan individual and multiple landscape management actions, but also how we evaluate and learn from their successes and failures (see e.g., [42]). In Scotland, as elsewhere in Europe, land managers are eligible for grants in return for implementing environmental improvement measures under Pillar 2 of the Common Agricultural Policy (CAP). A wide range of options are available relating to livestock, croplands and vegetation, among others (see e.g., [43]). Choosing the most appropriate measure for a given land holding is a complex problem involving multiple choices, actors and possible outcomes.

4. Methods

In order to respond to these three research questions, we conducted research and stakeholder engagement activities in two phases where Phase 1 addressed the first and second research questions and Phase 2 addressed the third. Future work will be addressed in Phase 3. The conceptual design of the process is shown in Figure 1, in which the specific actions described in this paper are numbered. These are described in Section 4.2 and Section 4.3.

Figure 1. Conceptual process design for stakeholder-driven development of an outcomes-based Environmental Decision Support System (EDSS).

4.1. Ethical Statement

All stakeholders gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, a widely accepted worldwide standard intended to ensure ethical conduct in scientific research on human subjects and approved by the Scottish government rural and environment science and analytical services division (RESAS) (project code: SRP RD1.4.3d). The ethical code followed was the “Research Ethics Policy for Human Participants” of the James Hutton Institute.

4.2. Phase 1: Stakeholder Engagement and Review of Software Options

Following a review of outcomes-based logic modelling ([37], 1.1 in Figure 1) adaptive management ([44], 1.2 in Figure 1), meetings were held with policy stakeholders in Scottish Government to discuss how logic modelling could support their outcomes-based approach in the National Performance Framework [41]. To start the co-construction of an outcomes-based logic modelling approach to aid decisions about landscape interventions to deliver multiple benefits e.g., water quality, terrestrial biodiversity and land manager income, a series of interviews were carried out with 13 national and regional level stakeholders about adaptive management and outcomes-based approaches ([45], Figure 1: 1.3). The results of this work are extensive and lie beyond the scope of this paper, but provided a useful starting point for understanding the rationales behind the outcomes-based approach.

Using initial insights from stakeholders of their needs to be addressed, experiences from colleagues involved in participatory GIS projects and our expertise in integrative modelling with stakeholders we started to narrow down the wide range of software options (1.4 in Figure 1). Potential software options were selected for their ability to meet seven initial requirements which formed our screening criteria (Table 1). Review of existing software options for supporting outcomes-based environmental management was undertaken from published, unpublished and internet sources. Where possible, we downloaded software and experimented with it in order to better inform our review. A short list of software options was then explored and an interactive prototype was developed to be demonstrated to stakeholders at the workshop.

[ Table omitted. See PDF. ]

4.3. Phase 2: Participatory Workshop

In Phase 2, we organized a half-day workshop with stakeholders with expertise in outcomes-based environmental management, in order to deepen understanding of requirements and begin a process of co-development (2.1 in Figure 1). The workshop was attended by six participants representing key domains in land and water management in Scotland, comprising representatives from the Cairngorm National Park Authority, Scottish Natural Heritage and Scottish Environmental Protection Agency, and three researchers with >50 years collective experience of developing practical tools to support land and water management for multiple benefits.

The workshop was based around a series of linked activities. In the first activity the participants were asked to rate a list of 17 “needs” for outcomes-based environmental management from information provided by stakeholders during the earlier interviews [45]. Participants were also asked to add additional needs that they felt to be necessary. For the second activity, working in pairs comprising a researcher and a practitioner, participants were invited to provide suggestions for the key principles that our approach would use to address the identified needs. The idea of “principles” is central to UCSD (see e.g., [31,46]) and is distinct from users’ functional requirements or needs in that it specifies how a system should be developed, not what the system aims to do. Following other authors, see e.g., [18,47] we developed a draft list of guiding principles for development based on own understanding of what was needed (Figure 2).

Figure 2. Researchers’ own draft principles, based on literature.

A sheet of existing principles from related approaches or applications was provided to the participants as a guide. As we were interested to see what principles the participants would come up with independently, we did not share our draft list. Principles suggested by the participants were discussed by the whole group, and at the end of the activity, the participants were invited to comment on the draft list of principles we had developed ourselves prior to the workshop.

The third activity included an overview of hands-on advanced testing of software options, and an overview of the process of development of the interactive digital prototype. The last activity involved facilitated discussion around the prototype on a large touch table. The workshop ended with a general discussion of the main themes explored. The outcome of the workshop involved production of a report [48], which was circulated to participants for their approval, which they were happy to give. The outcome of Phase 2 was the draft approach and prototype application (2.2 in Figure 1).

Audio and video recordings were made and photographs were taken with the full agreement of all participants. A wide range of material was collected from each activity including pro-formas completed by participants for each activity including commentary on post-it notes, a list of principles and detailed documentation of the discussions, recorded on wall-charts by facilitators during the workshop. A report containing all of this information was circulated to workshop participants who approved its content. A selection of this information forms the basis of the results section of this paper.

5. Results

5.1. Phase 1

The stakeholder interviews confirmed the need for tools that supported an outcomes-based approach with a central role for facilitation in place-based studies. There is a wide range of software tools and applications that are or could be used to aid facilitated discussions of place-based studies. Using the list of screening criteria (Table 1) we reduced a long list of 19 software options to seven for further hands-on testing and development. The long list of software options were classified into four broad groups. These were: existing adaptive management logic modelling applications e.g., Miradi (https://www.miradi.org/), existing desktop participatory modelling applications e.g., QUICKScan (http://www.quickscan.pro/), GIS applications e.g., QGIS (http://www.qgis.org/en/site/), and a range of software packages for developing web based applications e.g., R Shinydashboard (https://rstudio.github.io/shinydashboard/). In order to review these options in a structured way, we developed a set of seven criteria. These criteria were informed by our own previous work in this area, the interviews with stakeholders [45] and conversations with colleagues. These criteria are more a result of the researchers’ reflections following early conversations with stakeholders, than the product of detailed analysis. Nonetheless, they comprise a useful starting point for the development of the activities described in this paper. These criteria were as follows:

The tool/software:

1. Should be free at the point of use. This is an essential pre-requisite of the outcomes-based approach, which requires transparency and effective exchange of knowledge and data between stakeholders at all points in the process chain. Commercially licensed software is typically not free to use or share and thus cannot meet this basic requirement.

2. Should work on touch devices like mobile phones, tablets and larger touch tables. Touch-enabled devices are now very widely used, which is of itself an important reason why their use should be considered for EDSS. Touch user interfaces are popular because they facilitate user interaction, and, on larger devices, allow multiple stakeholders to interact at the same time. To ensure that any application can be widely shared, it should function on a range of devices, but without touch capability, the EDSS is restricted to ordinary PCs. At the same time, touch-enabled applications can easily be used with a keyboard and mouse if necessary, but standard applications frequently do not work on touch user interface devices. At present, many applications that have potential for use as EDSS (e.g., QGIS) are not designed for touch screen use, and we found that these tended to function poorly on touch devices. While a large touchtable is ideal for facilitating knowledge exchange between multiple participants (i.e., as in our workshop), outdoor workers, e.g., in forestry or agriculture, are likely to prefer an application which can run on portable devices like smartphones or tablets. In general, applications that are responsively designed to run on an ordinary smartphone may often be suitable for other devices as well.

3. Should have map-based functionality for users to interact with spatial information e.g., information on fields and other features related to land and water management. The focus on spatial information is crucial, since nearly all environmental decision-makers use maps as a means of communicating information about interventions in the landscape. At the same time, to establish a genuine process of knowledge exchange, information needs to flow in both directions, so some form of user interaction is necessary, either in terms of allowing the user to pull data from the system, or in actually offering the possibility to upload and work with users’ own data.

4. Should include functionality for outcomes-based logic models i.e., linking land management to a range of outcomes. This apparently very specific requirement can be seen to have broader implications for environmental decision-support. The software needs to be able to link the information provided to the user to desirable or required outcomes in a dynamic way. In this way it is possible to understand the implications of different interventions—e.g., for example, by adding a connection between land use type in a holding and sediment or nutrient run-off, a user would potentially be able to evaluate the likely outcomes of land use changes in terms of water quality and other policy and management objectives.

5. Should allow developers and end-users to develop and extend the software/existing application. The ability to modify and extend the software to add or change functionality was thought to be important, since environmental management is a dynamic sphere of action in which new datasets, plans and policies and strategies are frequently emerging. It should be clarified that we did not anticipate that all users should want or need to do this, but that it should in principle be possible for the development team.

6. Should have potential for scientific innovation. This is a constraint that may have little interest from the point of view of an end-user, but is important for researchers in today’s competitive science environment. A tool or approach that cannot show some potential to advance scientific practice or understanding cannot be expected to hold the interest of the scientific community, and makes it more difficult to obtain funding. It should be emphasized that we are not necessarily talking about technological innovation—finding new ways for stakeholder groups to interact around a problem would also be a worthwhile scientific innovation.

7. Should be actively maintained, preferably through a large, open user community. Given the previously mentioned interest in software that is free (at least at the point of use), it is important to distinguish between projects that are no longer actively maintained and those that are, preferable with a large or active user community. This is an important factor in ensuring the ability to modify or extend the software in the future—changes in PC hardware or software over time mean that software needs to be actively maintained. The multi-purpose software environment R and the popular free GIS package QGIS, for example, are actively maintained, with vibrant user communities, whereas the GIS software ILWIS, while still maintained, seems to be less actively developed and the user community, by comparison, is small.

Using these screening criteria we selected seven potential software options for more detailed hands-on testing (see Table 1). Six of these seven options were software packages and applications for developing web-based applications. The other option was ROAM which is QGIS application designed to aid field-based collection of data using touch devices. Software options were classified in four broad groups (Table 1), which are discussed as follows.

5.1.1. Existing Adaptive Management Logic Model Applications

The only software known to the authors in this group is Miradi, a software application intended to help adaptive management of conservation projects. Though there are a range of EDSS designed to operate within adaptive management contexts, e.g., the InVEST nutrient delivery model [49], these do not include explicit treatment of logic modelling approaches. Miradi was thus included as the only software explicitly designed to facilitate the logic modelling outcomes-based approach. Logic chain modelling is explicitly implemented through a flowchart view, and other functionality to assist work planning and budgeting and assessment of viability of targets. At present the software has no spatial component, but stated future plans for the software include development of a spatial mapping capability.

5.1.2. Existing Standalone Participatory Modelling Applications

This group comprises a small selection from the large range of modelling applications currently in existence that include explicit participatory modelling functionality or that have been used in participatory processes and is not intended to be representative or comprehensive (for more detailed reviews of participatory modelling software options and approaches see e.g., [50,51]). QUICKScan is a well-known commercial software application developed by Alterra, at Wageningen Netherlands, which is widely applied, including in Scottish cases [52]. It offers a rule-based approach to help stakeholders negotiate trade-offs in resource management contexts, and is explicitly spatial. This seemed quite appropriate for our case, but as a non-modifiable and non-free software, could not be included in our selection. Metronamica is another software application with 20 years + development behind it, very popular in the land-use modelling community. Though it does not offer specific participatory functionality, it is frequently used in decision-support contexts (e.g., [1]), and for working with stakeholders [35,53], even if its complexity means that stakeholders rarely use the model themselves.

5.1.3. GIS Modelling Applications

This group includes very well-known commercial applications like ArcMap as well as the popular open source application QGIS. These have been included because there are many spatial decision support applications built around these platforms, see e.g., for ArcMap [19,54] and for QGIS [55,56], but unfortunately, we found both of these to be poorly adapted to touchscreen environments, both in terms of operation (sluggish and buggy) and in terms of the user interface (tiny buttons, missing right mouse button functionality). The 3D Community Viz Public Participation GIS plugin for ArcMap was unusable on our touch table. The ROAM application, which provided a touch-screen friendly interface for QGIS was much more promising, and was selected for further investigation.

5.1.4. Software Packages and Applications for Developing Web-Based Applications

This last group stood apart from the other three software groups in that all of the tools reviewed in this group were web-based, and many were designed for mobile devices as well. Though these tools have the potential disadvantage of requiring adequate internet coverage, they can be designed to work off-line (e.g., progressive web applications). They represent a significant innovation in terms of user interaction and inter-platform flexibility. For all of the other groups, non-Windows operating system users are disadvantaged (with the notable exception of QGIS), and mobile device support is nearly non-existent. This group of applications stood out as addressing much more closely stakeholder’s needs, e.g., mobile, touch-enabled and Free-and-Open source (FOSS). With these applications a clear trade-off could be identified between being able to use “as-is” and customizability—in many cases there seemed to be good potential to flexibly incorporate and link maps, tables and charts. One application in particular, Rshiny, and its dashboard extension, Rshinydashboard, was very promising in this regard, since these tools are part of the well-known FOSS environment “R”, which is very popular with scientific users and has a large and active user community. For this reason, the advanced prototype was developed in Rshiny.

5.1.5. Summary of Findings from Software Survey

Overall, we found that by applying general criteria arising out of consultation with stakeholders we quickly reduced a long list of options to a handful of choices. This is not a reflection of a general scarcity of EDSS per se, rather an indication that the criteria we applied are not common features of EDSS.

5.2. Phase 2

5.2.1. Stakeholders’ Principles for EDSS

Workshop participants identified 12 principles for development of a software application for supporting environmental decision-making to improve the management of land and freshwater resources in Scotland. These are shown in Table 2. The key points that emerged from this activity and the subsequent discussion are described as follows.

[ Table omitted. See PDF. ]

1. Updateability: This was regarded as very important due to the rapidly evolving nature of environmental policy and the need to be adaptable and responsive. Participants emphasized the highly changeable nature of both the policy context and the data (Principle 1 in Table 2).

2. Practicability and usefulness: It was agreed that the tool should be both practical and useful. Since usefulness is subjective and therefore dependent on the end-user, involving end-users in the development process seemed like a good way to achieve this, e.g., through AGILE development principles [57] (Principle 2 in Table 2).

3. Clarity of purpose and objectives: Participants highlighted the need for clarity about the intended purpose and objective of the application. This is likely to depend on specific end users, further highlighting the need for a flexible, AGILE-type approach (see above).

4. Uncertainty, credibility and trustworthiness: Participants emphasized the need to deal honestly with uncertainty, both of data and of model/application outputs. This was regarded as having important implications for the credibility and trustworthiness of the information provided by the decision-support application (Principle 6 in Table 2). It was suggested that data should be filtered before providing it, since users may lack the knowledge to appreciate uncertainty. However, it was also appreciated that absolute certainty was not attainable, since scientists themselves frequently disagree about the trustworthiness of data or the reliability of the results. However, this was regarded as an important development principle even if it is not achievable in practice.

5. Openness about limitations/transparency: It was regarded as important to be transparent about what a tool can and cannot do. Transparency of the tool and of the decision-making process was a key principle (Principle 6 in Table 2). This is somewhat related to the issue of trust and credibility (above).

6. Specify intended scale of data and operation: In an issue related to the need for openness about uncertainty, participants raised the need to be clear about the intended cartographic scale of operation for the application. This can be partially addressed in practice by ensuring that datasets do not display at zoom levels that exceed their nominal scale.

7. Cost-benefit estimates for policy interventions: The need to provide cost-benefit information was highlighted by one of the participants, but others disagreed. One argument that was made against this idea was that it would be difficult to determine which actors would be best placed to make the cost-benefit assessment, since each actor would require a different cost-benefit analysis to suit their land/circumstances.

8. Diversity vs commonality: Although there was a high level of agreement about some aspects, for example, the importance of credibility (Principles 6 and 7 in Table 2) and accessibility (Principle 9 in Table 2), there was disagreement about others, e.g., appropriateness of cost-benefit analysis. This reminds us that end users are likely to have widely varying objectives in mind, and that it will be difficult to please all of them with a single application.

9. Decision support, not decision automation: Participants were keen to emphasise the need to be clear that the tool supports decisions, but does not actually make them.

In discussion, our own, more generalized principles (Figure 2) found general agreement. Several of our own suggested principles were also included by the participants, e.g., transparent, accessible and practical. However, participants observed that more clarity could be provided by setting them out as sentences (rather than as single words). Following the workshop, we drew up a revised set of general principles incorporating the principles suggested by stakeholders and following their advice (Table 3). This is intended as a synthesis, not a replacement, of the detailed principles set down in Table 2.

[ Table omitted. See PDF. ]

5.2.2. Facilitated Discussion around the Prototype Application

The prototype application (Figure 3), as we had hoped, since we had developed it in response to stakeholder criteria, was well-received. Though the application meets many of these requirements, being FOSS, optimized for touchscreen use and multi-platform, it does depend on the R software environment, which is not currently supported as standard on mobile phones, though it does appear to be possible to run it [58]. Stakeholders particularly appreciated the smooth map scrolling provided by the leaflet application, a GIS map server package available under R. The users appreciated the dashboard format, with a large map window occupying most of the screen and user controls and a linked chart output in a box to the right. The application was tested with land use data (Land Cover Map 2007 [59], which it allowed users to interact with (enquiry of land use type, clipping out a polygon and querying land use area in the polygon area). The land use data generated an interesting discussion, and stakeholders highlighted the importance of providing scale information, clearly naming the source of the land use data, and adequately informing about its known limitations. Participants were not widely agreed about the reliability of the national land use dataset, which led to a detailed discussion around reliability and trustworthiness of information presented.

Figure 3. Prototype application running on Windows. The screen capture shows the Water of Tanar sub-catchment of the river Dee in Aberdeenshire, Scotland. The user has drawn a polygon, which has been clipped out of the land use base map (LCM2007) and is displayed in the box to the right of the map window. Below it, the area of each land use in the polygon is displayed as a histogram.

6. Discussion

Outcomes-Based Environmental Management: What Do Users Really Want?

Our process is ongoing, and an exhaustive review of EDSS is beyond the scope of this paper. For this reason, our findings should be considered as propositions for discussion, not statements of certainty. Nonetheless, our work does suggest that some important aspects of EDSS design may often not receive adequate attention. The new breed of mobile and web-based tools offers new opportunities for application development, but at the same time, to develop software systems effectively it is necessary to understand user’s functional and non-functional requirements. This lies at the heart of producing useful software and is an established principle in software development that developers of EDSS could learn from. In order to understand user requirements, structured participatory processes like the one we have presented in this paper are necessary. The process of co-construction of an application typically passes through a series of key steps relating to facilitated participatory activities. Stakeholders can analyze their needs as environmental decision-makers, identify key development principles for a software application designed to address them and road-test an advanced prototype application.

We discovered that the tool our stakeholders wanted does not much resemble most published examples of EDSS used by the scientific community (see e.g., [60]). Our application should be free to use, touch and mobile-device enabled, part of an existing active development chain or community and be clearly oriented to understanding and evaluating environmental policy options in terms of outcomes, e.g., showing how a riparian buffer strip can improve water quality in a given location, or showing how land use change may influence diffuse pollution. It will be intended for use, probably in the field, by land managers and regulators, e.g., farmers, protected area managers or utilities operators.

One possible reason why the EDSS that has been conceptualized by our stakeholder group is so different to existing EDSS found in the literature is because of the specific requirements of our study and its context, e.g., catchment management in Scotland, adaptive management framework, and emphasis on understanding effectiveness of interventions through outcomes. But these aspects are also recognized as important in environmental management further afield. Though adaptive management has some well-recognized limitations (see e.g., [61]) it is still widely regarded, at least in modified guise of adaptive co-management (ACM) as playing a valuable role in good environmental governance [62]. At the same time, outcomes-based approaches are increasingly seen as best practice in environmental assessment, decision-making and stakeholder engagement [63]. Therefore the apparent absence of appropriate tools probably does indicate a real need.

In fact, there seems to be a genuine mismatch between what environmental decision-makers like our stakeholders want, and what EDSS developers, who are typically scientists, tend to produce (Table 4). This impression is supported by other researchers in EDSS (e.g., [18]), and is relevant to the science-policy divide in general (see e.g., [64]).

[ Table omitted. See PDF. ]

There are four key reasons for this mismatch in general terms.

1. Problem-solving stage. The kinds of EDSS that help stakeholders understand complex interactions, explore future scenarios, or predict existing tendencies are generally not suitable, or really intended, for use directly by practitioners, but are rather intended to supply information to them in the form of reports and briefings based on model results. Generally, practitioners do not need or want to run land use simulations for 2050 or experiment with the effect of different types of forcing mechanisms on climate. An EDSS that a farmer might hold in their hand to allow them to explore a range of options for managing a river catchment belongs to a different part of the process at a later stage in the policy process. This kind of EDSS operates under the assumption that first type described is already embedded in the policy-making process. This type of EDSS belongs to a different problem-solving stage, where, in the words of [65], “we agree to stop worrying about generalities and focus instead on impacts, mitigation, compensation, and accommodation”. There is a fundamental need to better understand the requirements of users at the policy implementation stage.

2. Type of decision-maker and level of action. A related question is the type of stakeholder to whom the EDSS is directed and the level of action at which they are typically operating. Water regulators with responsibility for large land areas, for example, may have advanced scientific training, and want to make use of integrated hydrological models like SWAT [66]. Local level actors, like municipal planners, conservation volunteers, or farmers, are unlikely to have such specific training and will have radically different questions they need to answer. The hypothetical farmer referred to above is making decisions at the scale of the land parcel, unlike, say, an agriculture minister, who is interested to know, in broader terms, about land use in general to inform land use policy more broadly. Many EDSS found in the scientific literature are concerned with these higher level or strategic decisions, while EDSS directed at the scale of the land parcel tend to be under-represented in the scientific literature (though see e.g., [10]). One of the most important priorities for our stakeholder group was the provision of reliable scientific information in an accessible and digestible format, rather than solution of complex analytical problems, or estimations of long-term tendencies.

3. Technological development time-lag. The significant time lag between submitting research proposals and the award of funding, and between development of a model or system and publishing it in a journal means that the most recently published EDSS are based on conceptualizations of technology (if not actual availability) from three or four years ago. Given the pace of change in information systems development, there is a significant likelihood that EDSS of the kind that emerged from our participatory process are currently being developed but are poorly represented in literature. An interesting review of this topic in the context of rural development can be found in [67].

4. Change in the role of the scientist. As can be appreciated from Table 4, the type of EDSS conceptualized on the right requires the scientist to play a different, perhaps less glamorous role, than that of aloof, objective expert of popular imagination. To help achieve sustainability objectives in land and water management scientific stakeholders should focus on facilitation and knowledge transfer at the level of implementation of concrete actions, as well as knowledge production activities higher up the chain.

Designers of EDSS need to move away from developing tools for standard desktop software and systems and make full use of mobile and web-based technologies with which people are becoming increasingly familiar. Web dashboards, already popular as “business intelligence” tools, provide the user with a range of integrated graphical outputs, e.g., a map, a histogram and a data table and offer an idea of how such applications might look. FOSS tools like Rshiny and Rshiny dashboard which facilitate web-development from inside widely supported scientific software environments with active user communities (in this case, R), seems like a promising direction of travel.

Providing complex information under high uncertainty in an easily digestible format is a major challenge. Our stakeholders were clear that the limitations and level of uncertainty in complex scientific information needed to be clearly explained in any application, and that great care should be taken not to provide information out of context or at inappropriate spatial scales.

Finally, our results highlight the importance of salience, credibility and legitimacy (e.g., [68]) in the development of EDSS. The group had major concerns around the trustworthiness of the data presented in the application (credibility). A process of co-development can help to increase credibility, especially if the data behind the application are well-known to, or even provided by, the stakeholders themselves. This also enhances legitimacy, by helping stakeholders to take ownership of the process and the tool. We suggest, as the title of our article indicates, that finding out “what users really need” is essential to ensure salience (relevance) of EDSS to its potential users, and that co-construction of any such tool, through a process similar to that described here, is a useful way forward.

7. Conclusions

Developing software applications to support land managers decisions under outcomes-based approaches to adaptive environmental management requires better stakeholder engagement practices to understand their needs and development requirements, and a stronger focus on new digital tools and open software and data. We have presented an example of a stakeholder-centered process to develop one such tool. By focusing on understanding and managing outcomes of environmental improvement measures, a preliminary review based on initial requirements led to the rejection of a large range of currently available tools and software as unsuitable. Many do not run effectively on the mobile and web-based devices that land managers typically use, and most are proprietary systems that require an upfront financial outlay and do not allow users to modify the source code or customize the application to suit their own needs. These aspects restrict their use to mostly scientific stakeholders and do little to facilitate environmental decision-making at the scale of the land parcel. What stakeholders really need are mobile and web-based devices that are free at the point of use and flexible enough to be easily adapted to meet changing requirements and different user needs.

Under the late nineties and early noughties “out-of-the-box toolkit” paradigm, scientists sought to package their knowledge into marketable products and systems under the assumption that clients would materialize with fully-formed problems requiring solutions. Since then, the world has moved on. Scientists now understand that this kind of hands-off back-office knowledge delivery is ineffective; these kinds of tools are rarely used. Not only do land-managers not want to use a complex computational model, they need software that answers specific questions like “how much will it cost to build a riparian buffer strip along the river that bounds my land, and to what extent will this reduce surface run-off from my arable fields?”, or “what specific interventions would be most appropriate in this part of the catchment, and what incentives and regulations are likely to be relevant?” To provide better answers to these questions, scientific stakeholders need to play strong facilitation roles at all points in the development process, including problem-framing, stakeholder engagement, application development and use.

Acknowledgments

The research described was funded by the Rural & Environment Science & Analytical Services Division of the Scottish Government. We are very grateful to all of the stakeholders who participated as part of this process.

Author Contributions

Richard J. Hewitt and Christopher J. A. Macleod carried out the research described and wrote the paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

1. Kok, K.; van Delden, H. Combining two approaches of integrated scenario development to combat desertification in the Guadalentin watershed, Spain. Environ. Plan. B Plan. Des. 2009, 36, 49–66.

2. McIntosh, B.S.; Ascough, J.C.; Twery, M.; Chew, J.; Elmahdi, A.; Haase, D.; Harou, J.J.; Hepting, D.; Cuddy, S.; Jakeman, A.J.; et al. Environmental decision support systems (EDSS) development—Challenges and best practices. Environ. Model. Softw. 2011, 26, 1389–1402.

3. Holmes, J.; Clark, R. Enhancing the use of science in environmental policy-making and regulation. Environ. Sci. Policy 2008, 11, 702–711.

4. Winder, N. Successes and problems when conducting interdisciplinary or transdisciplinary (= integrative) research. In Interdisciplinarity and Transdisciplinarity in Landscape Studies: Potential and Limitations; Delta Program: Wageningen, The Netherlands, 2003; pp. 74–90.

5. Macleod, C.K.; Blackstock, K.; Haygarth, P. Mechanisms to improve integrative research at the science-policy interface for sustainable catchment management. Ecol. Soc. 2008, 13, 48.

6. Allen, C.R.; Fontaine, J.J.; Pope, K.L.; Garmestani, A.S. Adaptive management for a turbulent future. J. Environ. Manag. 2011, 92, 1339–1345.

7. Allan, C.; Stankey, G.H. Adaptive Environmental Management; Springer the Netherlands: Dordrecht, The Netherlands, 2009.

8. Rosenhead, J.; Mingers, J. Rational Analysis for a Problematic World Revisited; John Wiley & Sons: Hoboken, NJ, USA, 2001.

9. Shim, J.P.; Warkentin, M.; Courtney, J.F.; Power, D.J.; Sharda, R.; Carlsson, C. Past, present, and future of decision support technology. Decis. Support Syst. 2002, 33, 111–126.

10. Sharifi, M.A.; van Keulen, H. A decision support system for land use planning at farm enterprise level. Agric. Syst. 1994, 45, 239–257.

11. Matthews, K.B.; Sibbald, A.R.; Craw, S. Implementation of a spatial decision support system for rural land use planning: Integrating geographic information system and environmental models with search and optimisation algorithms. Comput. Electron. Agric. 1999, 23, 9–26.

12. Theobald, D.M.; Hobbs, N.T.; Bearly, T.; Zack, J.A.; Shenk, T.; Riebsame, W.E. Incorporating biological information in local land-use decision making: Designing a system for conservation planning. Landsc. Ecol. 2000, 15, 35–45.

13. Holling, C.S. Adaptive Environmental Assessment and Management; John Wiley & Sons: Hoboken, NJ, USA, 1978.

14. Simon, H. The New Science of Management Decision; Harper Brothers: New York, NY, USA, 1960.

15. Gorry, G.A.; Scott Morton, M.S. A Framework for Management Information Systems. 1971. Available online: https://dspace.mit.edu/bitstream/handle/1721.1/47936/frameworkformana00gorr.pdf (accessed on 5 October 2017).

16. Engelen, G.; White, R.; Uljee, I.; Drazan, P. Using cellular automata for integrated modelling of socio-environmental systems. Environ. Monit. Assess. 1995, 34, 203–214.

17. Engelen, G.; Uljee, I.; White, R. Vulnerability Assessment of Low-lying Coastal Areas and Small Islands to Climate Change and Seal Level Rise—Phase 2: Case Study St. Lucia. 1997. Available online: http://www.dpi.inpe.br/cursos/tutoriais/modelagem/software_demos/simlucia/SimLuciaManual.pdf (accessed on 5 October 2017).

18. Winograd, M.; Pérez-Soba, M.; Verweij, P. QUICKScan: A Pragmatic Approach for Decision Support in Ecosystem Services Assessment and Management. In Handbook on the Economics of Ecosystem Services and Biodiversity; Edward Elgar Cheltenham: Cheltenham, UK, 2013.

19. Flacke, J.; De Boer, C. An Interactive Planning Support Tool for Addressing Social Acceptance of Renewable Energy Projects in The Netherlands. ISPRS Int. J. Geoinf. 2017, 6, 313.

20. Voinov, A.; Bousquet, F. Modelling with stakeholders. Environ. Model. Softw. 2010, 25, 1268–1281.

21. Van den Belt, M. Mediated Modeling: A System Dynamics Approach to Environmental Consensus Building; Island Press: Washington, DC, USA, 2004.

22. Barreteau, O.; Antona, M.; D’Aquino, P.; Aubert, S.; Boissau, S.; Bousquet, F.; Daré, W.; Etienne, M.; Le Page, C.; Mathevet, R.; et al. Our Companion Modelling Approach. 2003. Available online: http://jasss.soc.surrey.ac.uk/6/2/1.html (accessed on 31 March 2003).

23. Reed, M.S.; Evely, A.C.; Cundill, G.; Fazey, I.; Glass, J.; Laing, A.; Newig, J.; Parrish, B.; Prell, C.; Raymond, C.; et al. What is social learning? Ecol. Soc. 2010, 15, r1.

24. Volk, M.; Lautenbach, S.; van Delden, H.; Newham, L.T.; Seppelt, R. How can we make progress with decision support systems in landscape and river basin management? Lessons learned from a comparative analysis of four different decision support systems. Environ. Manag. 2010, 46, 834–849.

25. Mackay, E.B.; Wilkinson, M.E.; Macleod, C.J.; Beven, K.; Percy, B.J.; Macklin, M.G.; Quinn, P.F.; Stutter, M.; Haygarth, P.M. Digital catchment observatories: A platform for engagement and knowledge exchange between catchment scientists, policy makers, and local communities. Water Resour. Res. 2015, 51, 4815–4822.

26. Borowski, I.; Hare, M.P. Exploring the gap between water managers and researchers: Difficulties of model-based tools to support practical water management. Water Resour. Manag. 2007, 21, 1049–1074.

27. Uran, O.; Janssen, R. Why are spatial decision support systems not used? Some experiences from the Netherlands. Comput. Environ. Urban Syst. 2003, 27, 511–526.

28. Bagstad, K.J.; Semmens, D.J.; Waage, S.; Winthrop, R. A comparative assessment of decision-support tools for ecosystem services quantification and valuation. Ecosyst. Serv. 2013, 5, 27–39.

29. Leffingwell, D.; Widrig, D. Managing Software Requirements: A Unified Approach; Addison-Wesley Professional: Boston, MA, USA, 2000.

30. Norman, D.A.; Draper, S.W. User Centered System Design; Lawrence Erlbaum Associates: Hillsdale, NJ, USA, 1986.

31. Gulliksen, J.; Göransson, B.; Boivie, I.; Blomkvist, S.; Persson, J.; Cajander, Å. Key principles for user-centred systems design. Behav. Inf. Technol. 2003, 22, 397–409.

32. Sørensen, C.G.; Fountas, S.; Nash, E.; Pesonen, L.; Bochtis, D.; Pedersen, S.M.; Basso, B.; Blackmore, S.B. Conceptual model of a future farm management information system. Comput. Electron. Agric. 2010, 72, 37–47.

33. Sørensen, C.G.; Pesonen, L.; Bochtis, D.D.; Vougioukas, S.G.; Suomi, P. Functional requirements for a future farm management information system. Comput. Electron. Agric. 2011, 76, 266–276.

34. Kaloxylos, A.; Groumas, A.; Sarris, V.; Katsikas, L.; Magdalinos, P.; Antoniou, E.; Politopoulou, Z.; Wolfert, S.; Brewster, C.; Eigenmann, R.; et al. A cloud-based Farm Management System: Architecture and implementation. Comput. Electron. Agric. 2014, 100, 168–179.

35. Hewitt, R.; Van Delden, H.; Escobar, F. Participatory land use modelling, pathways to an integrated approach. Environ. Model. Softw. 2014, 52, 149–165.

36. Margoluis, R.; Stem, C.; Swaminathan, V.; Brown, M.; Johnson, A.; Placci, G.; Salafsky, N.; Tilders, I. Results chains: A tool for conservation action design, management, and evaluation. Ecol. Soc. 2013, 18, 22.

37. Macleod, C.J.A. How Can Logic Modelling Improve the Planning, Monitoring and Evaluation of Policy Measures and Wider Interventions for Multiple Benefits? 2016. Available online: http://www.hutton.ac.uk/sites/default/files/files/RESAS124_O3_D3_1_v0_5Final.pdf (accessed on 5 October 2017).

38. Funnell, S.C.; Rogers, P.J. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models; John Wiley & Sons: Hoboken, NJ, USA, 2011.

39. Treasury, H.M. The Magenta Book: Guidance for Evaluation. 2011. Available online: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf (accessed on 5 October 2017).

40. Scottish Government. Getting the Best from Our Land. A Land Use Strategy for Scotland 2016–2021. 2016. Available online: http://www.gov.scot/Resource/0050/00505253.pdf (accessed on 22 March 2016).

41. Scottish Government. Scottish Budget Spending Review 2007. Chapter 8: A National Performance Framework. 2007. Available online: http://www.gov.scot/Publications/2007/11/13092240/9 (accessed on 5 October 2017).

42. Pahl-Wostl, C. A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes. Glob. Environ. Chang. 2009, 19, 354–365.

43. Scottish Government. Rural Payments and Services: Management Options and Capital Items. 2017. Available online: https://www.ruralpayments.org/publicsite/futures/topics/all-schemes/agri-environment-climate-scheme/management-options-and-capital-items/#32412 (accessed on 27 January 2017).

44. Macleod, C.J.A.; Blackstock, K.; Brown, K.; Eastwood, A.; Gimona, A.; Prager, K.; Irvine, J. Adaptive Management: An Overview of the Concept and Its Practical Application in the Scottish Context. 2016. Available online: http://www.hutton.ac.uk/sites/default/files/files/research/srp2016-21/RESAS_srp143_aD1_ReportOnRelevantAdaptiveManagementApproachesForScotland_v0.8Final.pdf (accessed on 5 October 2017).

45. Macleod, C.J.A.; Hewitt, R.J. Summary of research on developing a more integrated approach to land and water management using incentives and regulations for the delivery of multiple benefits: Exploring national and regional level stakeholder views and needs. Unpublished work. 2017.

46. Gould, J.D.; Lewis, C. Designing for usability: Key principles and what designers think. Commun. ACM 1985, 28, 300–311.

47. Ball, J. Towards a methodology for mapping ‘regions for sustainability’ using PPGIS. Prog. Plan. 2002, 58, 81–140.

48. Macleod, C.J.A.; Hewitt, R. Workshop Summary: Developing an Outcome-Based Approach for Understanding the Effectiveness of Interventions in Catchments for Multiple Benefits. Unpublished work. 2017.

49. InVEST. Nutrient Delivery Ratio Model. Available online: http://data.naturalcapitalproject.org/nightly-build/invest-users-guide/html/ndr.html (accessed on 5 October 2017).

50. Voinov, A.; Kolagani, N.; McCall, M.K.; Glynn, P.D.; Kragt, M.E.; Ostermann, F.O.; Pierce, S.A.; Ramu, P. Modelling with stakeholders—Next generation. Environ. Model. Softw. 2016, 77, 196–220.

51. Basco-Carrera, L.; Warren, A.; van Beek, E.; Jonoski, A.; Giardino, A. Collaborative modelling or participatory modelling? A framework for water resources management. Environ. Model. Softw. 2017, 91, 95–110.

52. Dick, J.; Verweij, P.; Carmen, E.; Rodela, R.; Andrews, C. Testing the ecosystem service cascade framework and QUICKScan software tool in the context of land use planning in Glenlivet Estate Scotland. Int. J. Biodivers. Sci. Ecosyst. Serv. Manag. 2017, 13, 12–25.

53. Volkery, A.; Ribeiro, T.; Henrichs, T.; Hoogeveen, Y. Your vision or my model? Lessons from participatory land use scenario development on a European scale. Syst. Pract. Action Res. 2008, 21, 459–477.

54. Graymore, M.L.; Wallis, A.M.; Richards, A.J. An Index of Regional Sustainability: A GIS-based multiple criteria analysis decision support system for progressing sustainability. Ecol. Complex. 2009, 6, 453–462.

55. Novaczek, I.; MacFadyen, J.; Bardati, D.; MacEachern, K. Social and Cultural Values Mapping as a Decision-Support Tool for Climate Change Adaptation; The Institute of Island Studies, University of Prince Edward Island: Charlottetown, PE, Canada, 2011.

56. Wangdi, K.; Banwell, C.; Gatton, M.L.; Kelly, G.C.; Namgay, R.; Clements, A.C. Development and evaluation of a spatial decision support system for malaria elimination in Bhutan. Malar. J. 2016, 15, 180.

57. Cockburn, A. Agile Software Development; Addison-Wesley: Boston, MA, USA, 2002.

58. Teoh, T. Running R and RStudio from an Android Mobile Phone or Tablet. Available online: http://tteoh.com/technology/2016/10/20/r-rstudio-mobile-android/ (accessed on 20 October 2016).

59. Land Cover Map 2007. Available online: https://www.ceh.ac.uk/services/land-cover-map-2007 (accessed on 5 October 2017).

60. Grêt-Regamey, A.; Sirén, E.; Brunner, S.H.; Weibel, B. Review of decision support tools to operationalize the ecosystem services concept. Ecosyst. Serv. 2017, 26, 306–315.

61. Allan, C.; Curtis, A. Nipped in the bud: Why regional scale adaptive management is not blooming. Environ. Manag. 2005, 36, 414–425.

62. Plummer, R.; Baird, J.; Dzyundzyak, A.; Armitage, D.; Bodin, Ö.; Schultz, L. Is Adaptive Co-management Delivering? Examining Relationships between Collaboration, Learning and Outcomes in UNESCO Biosphere Reserves. Ecol. Econ. 2017, 140, 79–88.

63. Baldwin, C.; Hamstead, M. Integrated Water Resource Planning: Achieving Sustainable Outcomes; Routledge: Abingdon, UK, 2014.

64. Bradshaw, G.A.; Borchers, J.G. Uncertainty as Information: Narrowing the Science-Policy Gap. Conserv. Ecol. 2000, 4, 7.

65. Cartledge, K.; Dürrwächter, C.; Hernandez Jimenez, V.; Winder, N. Making sure you solve the right problem. Ecol. Soc. 2009, 14, r3.

66. Arnold, J.G.; Srinivasan, R.; Muttiah, R.S.; Williams, J.R. Large area hydrologic modeling and assessment part I: Model development. J. Am. Water Resour. Assoc. 1998, 34, 73–89.

67. Salemink, K.; Strijker, D.; Bosworth, G. Rural development in the digital age: A systematic literature review on unequal ICT availability, adoption, and use in rural areas. J. Rural Stud. 2017, 54, 360–371.

68. Cash, D.; Clark, W.C.; Alcock, F.; Dickson, N.M.; Eckley, N.; Jäger, J. Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making. 2002. Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=372280 (accessed on 5 October 2017).