Understanding the barriers to virtual student placements in the Semester of Code/Comprendiendo las barreras para el desarrollo de las prácticas virtuales de los estudiantes en el Semester of Code

Abstract

The Semester of Code initiative organised virtual placements for university students around Europe, working on authentic business problems using open source software. The project was welcomed by stakeholders, and many companies and open source foundations became involved. However, the response from students was disappointing. In this paper we examine the reasons for this, discussing the results of the evaluation work carried out. Finally, we consider the implications of our work for student placements and the Knowledge Alliance European Union programme.

Headnote

Abstract

The Semester of Code initiative organised virtual placements for university students around Europe, working on authentic business problems using open source software. The project was welcomed by stakeholders, and many companies and open source foundations became involved. However, the response from students was disappointing. In this paper we examine the reasons for this, discussing the results of the evaluation work carried out. Finally, we consider the implications of our work for student placements and the Knowledge Alliance European Union programme.

Keywords

Semester of Code; Virtual placements management; VALS project; Evaluation.

Recepción: 20-01-2016 Revisión: 10-02-2016

Resumen

La iniciativa Semester of Code organiza prácticas virtuales para estudiantes universitarios de toda Europa, trabajando en problemas reales de empresas que emplean software de código abierto en sus procesos de negocio. El proyecto fue bienvenido por todos actores involucrados, entre los que se encuentran varias empresas y fundaciones relacionadas con el software libre. Sin embargo, la respuesta por parte de los estudiantes fue menor de lo esperado. En este artículo se examinan las razones de ello, se discuten los resultados de la evaluación que se ha llevado a cabo. Finalmente, se reflexiona sobre las implicaciones del trabajo realizado para las prácticas en empresas de los estudiantes y el Programa Knowledge Alliance de la Unión Europea.

Palabras clave

Semester of Code; Gestión de prácticas virtuales; Proyecto VALS; Evaluación.

Aceptación: 29-02-2016 Publicación: 01-03-2016

1. Introduction

The VALS project was conceived to bridge the gulf in knowledge and skills between university students and the software industry which they hope to be employed. The vehicle for achieving this was the Knowledge Alliance programme of the European Commission. The proposition which informed project was set out in the project plan (García-Peñalvo et al., 2013), and is described in (García-Peñalvo & Cruz-Benito, 2016; García-Peñalvo, Cruz-Benito, Conde, & Griffiths, 2014, 2015; García-Peñalvo, Cruz-Benito, Griffiths, & Achilleos, 2015, 2016; García-Peñalvo, Cruz-Benito, Griffiths, et al., 2014). The approach is based on the following observations.

1) There is a need to develop practice in virtual placements. European higher education students need to be better prepared for the workplaces where they will earn their living, while businesses need more capable employees (García-Peñalvo, 2015, 2016). Physical placements of students in businesses depends on a network of local connections, the location of placements is restricted by the high costs of relocation and living expenses at any significant distance from the home institution. Virtual placements can overcome these limitations.

2) Existing practice with technology favours the development of virtual placements. Much of the European economy is now mediated by online communications (García-Peñalvo, 2011), and the solutions to many business problems involve the development of code. Consequently, there is an opportunity for HE and businesses to collaborate around technology projects. More specifically, open source software (OSS) is an enabling technology. It can provide a shared infrastructure: it is accessible to students, and businesses are not constrained by intellectual property or commercial interests, which prevent them engaging with educational placements. OSS also has well established collaborative practice within which authentic business tasks are shared remotely, and beyond the confines of an individual organisation. Moreover, OSS foundations are hubs, which channel the operational challenges of their users through to the people who can solve them.

3) VALS proposed that universities, companies and foundations could all meet important goals through collaboration on open source development, but that this opportunity had not been taken because of the lack of support for managing and promoting collaboration across the two sectors. Consequently, VALS set out to provide the methods, practice, documentation and infrastructure to unlock this potential, and to pilot this as the "Semester of Code", echoing the successful Google "Summer of Code". In this paper we refer to Google Summer of Code as 'GS°C', and to Semester of Code as 'S°C'.

The purpose of this paper is to consider the experience of the VALS project as a whole, and to draw conclusions which we believe are relevant and useful to other Knowledge Alliance projects, and to the wider communities involved in linking education and software development. In brief, the project found that there was:

* An enthusiastic response from the companies.

* Varied responses from Universities.

* A very limited response from students.

In this paper we offer a reflection on project activity, with consideration not only of the project outcomes, but also on what these imply for work placement programmes, and the preparation of students for the workplace.

The paper is directed at three types of reader:

1. Those interested in the relationship between education and technological development in the wider society, with a particular focus on open source software.

2. Those involved in Knowledge alliances (funders, managers, project workers, and participants). We believe that the VALS project has uncovered valuable perspectives on the relationship between educational institutions and their students, and work placements.

3. University managers and lecturers. The VALS project raises important questions for higher education about the preparation of students for the world of work.

2. Building support for managing and promoting collaboration

The description of work in the project plan set out in detail what should be done to achieve the project goals, and all actions specified were carried out. A workflow was developed to manage the placements, and guidance was produced for the different categories of participants (foundations, companies, universities, students). Dissemination activities were carried out to promote the programme, and coordination carried out between the various actors.

One important practical problem required a substantial change to the approach taken. It had been planned to use the open source Melange application, developed by Google to run the Summer of Code, but it proved impossible to deploy the system for VALS. Clearly Melange works as an internally deployed Google Summer of Code platform, but for third parties it suffered (in our experience) from

a) lack of a coherent policy for the public release of stable code;

b) development practice which is based on undocumented updates to the live code, which are only reflected in the code base in an ad hoc and retrospective manner.

The fact that the code only runs on Google's web application hosting platform makes it harder to deal with these issues as that platform can be considered as a black box. The project was left with no alternative but to develop its own virtual placement system (VPS), which was successfully achieved, as described in D3.1. and D3.2 project deliverables. For a discussion of the rationale for this decision, see (García-Peñalvo, Cruz-Benito, Conde, et al., 2015).

This decision to build our own VPS was justified at the close of the project by the news that Google had discontinued all work on Melange (Taylor, 2016), was archiving the Melange Web site, and moving Google Summer of Code activities to other websites.

The S°C was launched in August 2014, and extensive dissemination work was carried out. The results were as follows:

* Companies and foundations registered at the platform in more than sufficient numbers. They registered themselves at a mailing list, received a registration code (so that we knew only serious participants entered) to register themselves. In total 64 organisations inserted 238 projects.

* Plenty of dissemination, but the universities were lukewarm. As a start the Semester of Code was promoted in the universities that were part of the project: University of Cyprus, Salamanca and Università di Udine. In late autumn 2014 the list of universities was extended to include: Paris (France), Venice (Italy), Universitat Politècnica de Catalunya (Spain), Léon (Spain), Frederick University, Open University of Cyprus, University of Nicosia (Cyprus), University of Bolton (UK), University of Oxford (UK) and the University of Belgrado (Servia).

* Very limited response from students. We had 33 students registered and 11 students that submitted one or more proposals from which 7 were accepted and completed. We then extended the project to run a new instance from February on. We had the consent from the mentors and organisation administrators to reuse their project ideas again as only 7 were really carried out (out of 238). This second instance had a superset of the first set of students minus the students who had an accepted proposal. With that in mind we started with 59 students from which 21 new students submitted a proposal and the 4 students from the first instance were transferred. From these 25 proposals 10 were accepted, 2 rejected and 7 never rejected nor accepted. The others were never submitted.

This level of take-up among students was substantially lower than the project had hoped for, and was a puzzle to the project team. The partners thought that this the Semester of Code was a good idea, and the proposal evaluators of the European Commission agreed. When we presented the project proposal, the audiences were generally positive about the idea, and seemed to think that it could be successful. In informal discussions the usual response was "What a great idea!". But when it came to actually submitting projects, there were few to be seen. The problem addressed this paper, therefore, is to understand the ways in which our circumstances differed from more successful applications of the underlying approach, and to draw out the implications of this insight for knowledge alliances, and for education as a whole.

3. Evaluation

The S°C is a complex process, involving a three groups of actors, students, mentors and academics. These groups three were involved in new types of interactions, in many cases with people they had never met before, and they had no reason to consider the efforts of the project to promote and facilitate the S°C process, or the consequences of those efforts. The only people who were aware of the whole process of S°C were the project staff, and it is therefore to them that we look for theories about the dynamics which led to the results. These theories were developed in a whole day focus-group involving members of the project team, with follow up activities to expand on and interpret the information that had been gathered. These theories about dynamics of the project, if they are to be convincing, need to be tied to the experience of the participants in the project. Consequently, we also carried out an evaluation process with 80 respondents who were in one way or another involved in the S°C. In this paper we bring together these two lines of evaluation work, contrast them with the results of the project, and draw conclusions. We first consider the questionnaires.

3.1 Questionnaire

Three questionnaires were developed, for students, mentors and academic supervisors, and were implemented and delivered using the Google Forms platform. The details of the questionnaires and the results of the analysis are available in the VALS evaluation report (García-Peñalvo, Griffiths, et al., 2015). In this paper we highlight some of the results which are most relevant to understanding the differing enthusiasm for placements between students, supervisors and mentors.

The students were consulted in two phases. The first questionnaire examined engagement with the S°C, views of the placements on offer, and the experience of developing a proposal for participation. This questionnaire was made available in presentations about the S°C across the Universities involved in programme, and during the process of applying for virtual placements. 50 responses were received in this phase. The second phase was directed towards those students who had participated in one of the seventeen completed placements, with the aim of understanding their experience of the placement process and its outcomes. There were 12 responses to this questionnaire from a population of 17.

The mentors and academic supervisors were consulted in a single phase process in order to gather the perspectives of these two key groups of actors. The questionnaire to mentors received 13 responses, and the questionnaire to academic supervisors received 14 responses, both out of a population of 17.

A total of 89 responses was received to all the questionnaires.

3.2 Student pre-placement results

The student pre-placement questionnaire showed that of the students surveyed 40% had visited the website, 14% had taken the step of registering, but only, 4% (i.e. two respondents) had taken this forward to making a proposal. Both of these two proposals were accepted by mentors. If this rate of response were extrapolated to the 17 placements made, then this would mean that presentations to 25 students were required to obtain a single placement, and 425 presentations to achieve the 17 placements which resulted from the S°C programme. However, it should be remembered that the respondents who made the effort to answer this questionnaire were likely to be more favourably disposed to collaborate with the project than those who did not respond. These figures therefore indicate a lower limit, and the actual required rate of engagement is in all probability substantially higher. Be this as it may, these results show that the problems experienced by the project in obtaining commitment to a project were not due to a failure to present the project to potential applicants.

Those students who did not apply for a placement were asked why they had not done this. They were offered a choice of 10 explanations, plus the option to add their own explanations. Respondents could provide more than one explanation, and thus there is a total of 79 responses from the 44 students who answered this question. The students' own explanations are indicated as 'Other' in Table 1. In the table the explanations have been grouped in order to identify the key themes in the students' responses.

By far the most frequent explanation is that there is lack of time for carrying out a placement (47 responses, more than half of the total of 79). Given that there were 44 respondents, it seems that the great majority of respondents identified this as a problem. The followed by 'shortcomings in communication' (14 responses), lack of confidence (12 responses), and lack of motivation (10 responses). We will return to these themes in our later discussion.

3.3 Student post-placement results

Seventeen students who had participated in placements were included in this phase, and 12 responses were received. The students were asked to rate their experience on a five point Likert scale from 'very bad (1)' to 'very good (5)'. They mostly reported a good general experience (mode = 4), and a very good experience with their mentors and academic supervisors (mode = 5 in both cases), and would also recommend their host organisations for future placements. Students also felt that there was a good relationship between their university studies and their placement (mode = 5). Students were less clear on the relevance of the placement in planning their future careers (split mode = 3 / 5), and on the help which the placement would provide in finding a job (mode = 3).

These results show that the placements were largely appropriate for most students, and that, at least for those students who progressed to a placement, the S°C approach to placements is viable. However, 5 students failed to complete their placements. Three explained that this was due to shortage of time

* The time and communication between the tutor and I

* I had no time to develop it

* It's not completed but in progress, due to personal issues the development of the assignment was delayed for several months.

Two others ascribed their difficulties to lack of knowledge about the technical aspects, and in one case also to lack of support from their academic supervisor.

Ten respondents described their learning goals during their placement, almost entirely identifying technical issues Only one participant mentioned the kind of learning that can only be obtained on a placement: "Working on an already functioning product, integrating new functions on an existing architecture". The need to learn new technical skills calls into question the high ratings for the alignment between university studies and the requirements, mentioned above, and suggests that only the most confident of students would be willing to put themselves forward if they realised in advance this need. On the other hand, when asked what they had in fact taken away with them from the project, seven of the eight who answered the question mentioned some aspect of collaboration, open source communities, and project management. Thus, those students who were able to take on challenging technical tasks did experience a benefit beyond those specific tasks.

3.4 Mentors

The students were happy with the mentors: of the twelve 6 gave the mentor the maximum score, and none the lowest score). Among the mentors however, four rated their experience of the students as 'bad' and three as very bad, although 6 were rated as 'very good'. Given that five projects were not completed at the time of data collection this result is not surprising. Nevertheless, it indicates the kind of judgement which students might be reluctant to voluntarily submit themselves. However, the mentors were more positive than negative in response to all questions about students, in all areas, including task management, communication skills, responsibility, adaptability, creativity, personal involvement, motivation. receptiveness to criticism, punctuality and relationship with the work environment. There is therefore not a significant mismatch between students' capabilities and the demands of the placement.

The mentors identified a number of issues with the organisation of the S°C programme. One noted the lack of a company or organisation backing up the programme (other than the project coordinators). Some commented on the unclear deadlines which resulted from the looser arrangements in the second phase of the programme. Two mentioned shortcomings in the Virtual Placement System interface. However, 12 out of 13 respondents said that they enjoyed the experience of mentoring, and 9 said that they would participate again, so these problems were clearly not severe.

The mentors made suggestions for improving the S°C. Of the eleven suggestions, six suggested making payment to students to increase their motivation. The remaining suggestions requested a tighter schedule and/or greater clarity in the communications between the S°C programme and the mentors.

3.5 Academic supervisors

Like the mentors, the academic supervisors (hereafter 'supervisors) rated the students as being positive in all categories. The positive view was, however, substantially stronger than that shown by the mentors, with only one student rated "very bad" in one category (punctuality). All supervisors enjoyed taking part, and would repeat their participation, again showing themselves to be more satisfied than the mentors. These results again suggest that the placement environment is a harsher test of student's abilities and performance than is academia. If students are aware of this, it may well discourage them from participation in S°C placements.

The problems experienced by the supervisors were primarily related to the complications of making agreements with institutions (eight of thirteen comments). Four comments related to the difficulty of contacting mentors, and one concerned the features of the VPS.

In describing what they had learned, seven of twelve responses chose the statement "Administrative processes in the universities are very strict, especially regarding schedules, and are not shared by all the universities all over Europe. Schedules and workload in companies are not always compatible with universities ones and students' planning". One supervisor chose the related "Schedules and workload in companies are not always compatible with universities ones and students' planning".

The improvements to the S°C process are aligned with these comments. In eight of thirteen responses supervisors chose options requesting greater flexibility in the establishment and execution of student projects, while three requested greater alignments with university curricula and processes. This is a strong indication that the rigidity of academic processes is a barrier to the S°C approach.

In explaining non-completion, three of the seven comments focused on lack of responsiveness from mentors, two on student performance, and two on personal circumstances. Supervisors made very similar comments on the problems experienced by students.

4. Focus group

A focus group of project participants undertook an analysis of the factors and mechanisms related to the outcomes of the project, both those favouring the project and those militating against its success. The factors were gathered from ten project staff at an all-day workshop, and transcribed into a spreadsheet with the following attributes:

* Topic / grouping (free text: identification of explanatory factors).

* Level of analysis (fixed values: basic observation, interpretation, recommendation).

* Difficulty (fixed values: easy, hard, intractable).

* Locus (fixed values: Student, University, Business, Commission, Society).

* Theme (free text).

The result was a collection of 73 items, Subsequently the evaluation team went through the collected texts, and after two iterations of analysis categorised them into the following categories:

* Responsibility and autonomy.

* Self-confidence.

* Students need for guidance.

* Process and communication.

* Curriculum.

We now discuss these in turn, synthesising the items in each category. The results do not constitute a characterisation of the institutions involved, but rather seek to capture the theorising about the success and failure of S°C of a group of people who had worked with their colleagues to promote the S°C for two years.

4.1 Responsibility and autonomy

4.1.1. Analysis

In many ways virtual placements are more demanding than face-to-face ones.

* When undertaking virtual placements students need to be able to operate more independently than in face-to-face placements, and need a stronger personal commitment because there is nobody to remind them or complain if they do not carry out their tasks.

* In virtual placements the only activity is to carry out a real task with real code. There is no tea making or photocopying. In a face-to-face placement, on the other hand, there are many introductory tasks that can be given, with no real responsibility or professional development.

* Face-to-face placements are often set up a year in advance, requiring that the tasks are to some extent set aside from the main flow of work in the company. Part of the benefit to the company may lie in the opportunity to get a close knowledge of a potential employee. In S°C, on the other hand, it is expected that the task to be undertaken will be one that the company genuinely wants to have done (although not necessarily a critical one).

* In S°C placements there is a good chance that students will be working for leading technology companies or foundations, while in local placements this is less usual.

The demanding nature of virtual placements is not in itself a drawback, as the independence which they require is a reflection of the professional reality which many students will meet when leaving university. The challenge which virtual placements represent does, however, reveal any shortfall in a student's capability, motivation and self-confidence more starkly than a face-to-face placement. Similarly, the personal and professional skills involved in planning and completing the work are put to the test more rigorously in virtual placements. The S°C initiative is not of a sufficient scale to draw conclusions about the whole of the European Higher Education area. Nevertheless, the consensus drawn from conversations with lecturers and in project meetings is that Universities have increasingly come to adopt the methods of schooling, providing programs that do not stimulate autonomy as much as they may have done in the past (with the potential for drop-out and failure that this autonomy brings with it).

4.1.2 Recommendations

It seems that students require more support in the virtual placement process then the VALS project foresaw. This is strongly related to the 'guidance' theme which we discuss below. Lecturers could prepare students by spending a lecture on what to expect both in terms of expectation/communication and tooling. Additional support could be provided for the writing of a proposal by providing templates or good examples. However, such examples constitute a pedagogic intervention as well as a support, and can only to a limited extent can this need be covered by the S°C Association. More generally, the experience of S°C indicates the need for universities to focus explicitly on the development of professional responsibility and autonomy, and to address the question of whether the increasing element of schooling in HE is restricting the ability of students to acquire these capabilities.

4.2 Self confidence

4.2.1 Analysis

The experience of tutors, passed on to project staff is that many students did not want to participate in S°C because they did not feel confident that they would be able to perform the tasks satisfactorily. This was also a theme in the results of the pre-placement questionnaire, although by some distance less significant than the lack of time. One might, moreover, wonder whether students a reluctance to admit to a lack of confidence may have reduced this score. For some students this was in part a matter of technical expertise and experience of real-world development. In some countries, however, a perceived lack of linguistic skills in English was a common barrier. They were concerned that they would not fully understand the mentor or the community, and that they would not have the vocabulary that they would need to express themselves clearly. In some countries, for example in Spain, the general level of English among students is so low that there is no stigma attached to using this as an excuse for not participating in S°C. In Germany or Holland, for example, the situation is quite different.

As we discussed in the section on the Curriculum, this problem is much more evident in virtual placements than in face-to-face placements, as in the latter the social aspects of the workplace can cover up any underlying difficulties.

4.2.2 Recommendations

The solution to the problem of self-confidence lies beyond the scope of the S°C. The experience of running the S°C, however, does clearly show that the level of confidence which students have in their ability to carry out the professional tasks for which they are training leaves much to be desired. We encourage universities to pay greater attention to developing students' confidence in their ability to develop and submit proposals, as discussed in the section on Curriculum. We note that that the lack of a good working knowledge of English which we have found in a number of contexts is a severe barrier to both virtual placements and professional practice, and universities should take this issue seriously.

4.3 Students need for guidance

4.3.1 Analysis

The S°C offers little help with the complexity of managing the different student abilities and attitudes on the one hand and project requirements on the other. In this S°C is following the example of GS°C, but our evaluation and experience suggests that this raises serious problems. GS°C has achieved the critical mass which is needed for their approach to work. This enables GS°C to resolve the complexity of matching by having large numbers on both sides: there are a lot of projects to choose from, and a large number of competing proposals many of which get rejected in the end. In Google Summer of Code there may be so many students that organisations can filter the proposals. In S°C, in contrast, where there tended to be only one proposal for a project, then mentors tend to accept that proposal, and there were only two rejections in the pilots.

However, there is an underlying difference between GS°C and S°C. Google can adopt their position with regard to students, partly because they have no responsibility for the students' future, and partly because GS°C carries no academic credit and so rejection of a project has no consequences to students other than a bruised ego. They are, after all, organising a competition with a monetary reward, and their position follows the practice of a competition. The situation is different for universities, which do have responsibilities to students, and have an interest in their longer term future (both for ethical reasons, and in order to improve the institution's reputation). A failed project will have some impact on the student's performance. Although this can be minimised by alternative assessment processes, universities cannot allow students to fail too much. This places a much greater importance in S°C to the matching students to suitable projects, so that they do not fail. In any event, in practical terms S°C cannot take a filtration approach to the selection of projects, because the numbers of proposals have not been sufficiently large.

In view of these considerations, the choice of project by a student in S°C is more significant than it is in GS°C. However, feedback from tutors and students, and the results of evaluation, indicate that students are often unsure if they will be able to carry out the projects. In this situation, they do not have anyone to advise them. There is functionality in the S°C VPS which enables lecturers and tutors to filter the projects and mark projects as being suitable for their students. However, this functionality was not used, because supervisors do not have detailed knowledge of the projects available, and may not have been active in the OSS world for some time. Nor, in many cases, do universities have a clear picture of the capabilities of their students, beyond the limits of their performance in tests. They are therefore not in a position to make recommendations. In the evaluation 10 respondents out of 50 commented that there were Insufficient details on Semester of Code for them to take their decision. It is true that the quality of the descriptions varies (as it does in GS°C) but we believe that the underlying problem is not that descriptions could be improved, but rather that the students have no other guidance available.

4.3.2 Recommendations

The circumstances of students, their capabilities and concerns vary enormously between institutions and across Europe. It is therefore not practicable for the S°C Association to attempt to provide guidance in matching students and projects, or to seek to address their specific difficulties in carrying out placements. The general guidance materials about the S°C can of course be revised, and this should be done on a regular basis. But this does not address the key problem of matching students to projects. It would be valuable for S°C to support the writing of clear project descriptions through editing, and by considering a revision of the template. However, the Association holds no leverage over mentors, and so cannot make demands. Participating HE institutions should also be alerted that some engagement of academic staff in the project matching process is highly desirable.

Because the problem of guiding students in finding projects is inevitably local, there is no way of avoiding the need for the education institution to take on some responsibility. We believe that in all placements it is necessary for university staff to have a close engagement with the domain in which students will be working, and for staff to have a better understanding of their own students. The exigencies of virtual placements make this clearer than face-to-face placements. To address this need, there is a need for engagement between OSS foundations and companies and the university, so that universities and students understand what is required for contributing to code, and organisations have confidence in contributors (and in particular contributors from a specific institution). University research groups or departments should be encouraged to take on a role in specific open source projects, and create an open source culture. It would also be valuable to involve student organisations in promoting participation in open source projects. These measures would provide tutors with the knowledge necessary to judge projects and students, and to help in recommendations. In the longer term a relationship could be built up between departments and particular foundations, which would greatly facilitate the process. Successful examples of this approach are Oregon State University Open Source Lab (Casson & Hawthorn, 2011) and NCSU Open Source Initiative (2014) in North Carolina State University.

4.4 Process and communication

A number of issues can be grouped together under the heading of process and communication.

4.4.1 Differing expectations

4.4.1.1 Analysis

Industry often requires an agile response from students making proposals, and from students who are assigned projects. This agility is often difficult to satisfy due to university inflexible schedules and enrolment procedures. The organisations hope for a response in weeks, but by the time the students are on the course it is months later. There is a period for organisations to register which could be aligned with the availability of students at some university but unfortunately the curricula of the universities are not aligned at all. The S°C therefore finds itself trying to satisfy conflicting expectations. Because of the varying university time schedules and the need to find suitable students, we loosened the original tight time schedule in the platform in the second instance of the pilot. However, this runs the risk of frustrating project owners, who want their projects to be addressed soon after they post the description. These conflicting interests are reflected in the questionnaires, in which a number of supervisors request flexibility, while mentors request clearer deadlines.

4.4.1.2 Recommendation

It would be desirable to have a means of coordinating expectations, by means of agreement on a flexible way on the planning of a Semester of Code instance. It could be either totally open (leaving it up to the student and to the organisation when to enter) or it could have a fixed schedule of periods in which certain matchmaking processes occur. However, if the placement programme is still in the building stage, then there will not be critical mass, and if the students do not materialise at the expected date, then confidence in the programme may be undermined. To enable this coordination of expectations, we anticipated on future uses of the VPS by making it possible in the VPS to offer a project idea for a certain time. Projects can be enabled/disabled by the project owner and setting start and end dates showing the availability of a mentor for this project. To make this work, mentors are encouraged to make this period as long as possible to let the loose matching approach keep its strength.

4.4.2 Mutual commitment

4.4.2.1 Analysis

In some cases, there were difficulties in maintaining communication with industry mentors. This was particularly the case when the period between publishing the project and the receipt of a proposal was relatively long. While the results of the questionnaires indicate that the experience of supervisors and students with mentors was generally very good, some isolated problems were experienced. Over a number of months, the mentor may have moved on to other activities, or even another job, and communication may become difficult or impossible. On other occasions there may simply be a lack of full commitment from the industrial side, which the S°C has no authority to remedy. Examples of this are the occasional proposals that never got a definitive yes or no from the mentor, and the lack of a formal sign-off in the VPS for some of the projects, although they were actually finished.

4.4.2.2 Recommendation

S°C should support the process of signing of a kind of contract at the beginning of the project (after the match is made between student and mentor and supervisor). The VPS supports this kind of contract it is not enforced. It would also be valuable to sign he project off, in the system when it is completed. This is supported in the VPS , but here also, there is no means of enforcing this. If students received their credits upon finalising this agreement in the system (beforehand and after the finish of the project) there would be a strong incentive for both students and tutors (and indirectly also for the mentor). There are three important advantages of that:

* It is easier to monitor the S°C process, and the number of finished projects, and the time the student has taken to complete them.

* The sign-off process is an ideal opportunity for undertaking evaluation

This recommendation, however, creates a link between S°C processes and academic processes which will require an increased institutional commitment to S°C.

4.4.3 Process

4.4.3.1 Analysis

The S°C Association has a substantial asset in the Virtual Placement System (VPS) developed by the project. Using the VPS, it is easy to establish an instance of the S°C, means that the association (or anyone else) can in future avoid the difficulties experienced by the VALS project with Melange. In response to requests from participants, the VPS is able to run an instance in a very flexible way, for example with a loose schedule lasting a year, or to establish and enforce a predefined time-line designed for that instance. Documentation is available for setting up a new instance, and for carrying out the relatively light load of maintenance.

In general, the process supported by the VPS has been satisfactory, but three aspects have been identified which could be improved. None of these, however, has been a major obstacle for S°C. For all three aspects the technical adjustment to the workflow is a relatively minor problem. Rather the challenge is to understand, or change, the practice of participants.

1. The VPS process assumed that bringing together students and organisations on one platform would lead to matches between participants with a need for practical experience (students) and participants with a need for a specific software task to be carried out (organisations). Experience has shown that more facilitation is needed (as discussed in the section on 'Guidance'), but that lecturers and tutors do not help their students by using the functionality already available for rating projects. The VPS provides some feedback to students and mentors on the status of projects, the number of proposals etc. and a mailing system is in place to send that information to the relevant people, but more information could be provided to support lecturers and mentors in their guidance of students.

2. The strategy of limiting the system to the organisation of the placement, rather than also supporting the placement activity itself, appears to have been correct. The S°C strategy is that placements need to take place using the authentic systems of the particular workplace, and in no cases has this been a problem. As discussed in the 'Mutual commitment' section, there is a need to encourage users to return to the VPS once the project work has been completed, which will involve some minor changes to the system.

3. Some partners had particular needs which had to be handled in an ad hoc manner, rather than being part of the S°C process. For example, in some universities a contract has to be signed with organisations providing the projects. These additional workflow processes could be included in the VPS.

4.4.3.2 Recommendation

In all three aspects, the provision of support in the workflow only makes sense as a response to a solution identified in discussions between supervisors, mentors and institutions.

Maintenance of the VPS is a light task, and no major changes have been identified as being necessary. Nevertheless, somebody needs to have the time and money to spend time on these tasks. This person should also keep track of projects/mentors that are no longer supported/relevant. This task will fall in the first instance to the S°C Association, and some additional administrative functionality could be added to the VPS (for example making it possible to email a project without having to manually find the responsible persons address).

4.5 Curriculum

4.5.1 Analysis

International collaborations, and more specifically the S°C proposals and procedures, fit well in the culture of companies related to OSS and the communities that support it. This is borne out by the substantial response to S°C from those communities. However, from the experience of S°C, and from the response of supervisors to the questionnaire, it seems that in universities there is little culture of cross-institutional collaboration in undergraduate teaching. This applies at both national or international levels, although the extensive networks of international research activities may serve to mask this lack. Indeed, the situation may be getting worse in response to changes in the environment in which higher education operates. As Choy and Delahaye (2009) state about "Work Integrated Learning (WIL)", successful WIL is premised on a learning partnership where the power over the curriculum and pedagogy are shared". The conditions for this sharing, are being constrained in European education, while the climate of innovation and creativity necessary for curricular innovation is under pressure. According to the focus group, a number of trends are converging which, perhaps counter-intuitively, have a chilling effects, as we now outline.

* There is increasing pressure on universities to meet the needs of industry and society, as argued by the University Alliance (Ansell, 2016). This has led in many cases to the addition of content to the curriculum, with increased pressure on time. See also the responses of students to the pre-placement questionnaire, above, which also stress the importance of this factor.

* The trend towards evidence-based educational policy (Missett & Foster, 2015) means that national governments are increasingly demanding greater evidence of effectiveness in the way that university education is carried out. Student activities in a placement are harder to document and certify than activities inside the institution.

* There is increasing monitoring and validation of courses and curricula. This makes it harder for innovative solutions to be introduced, and increasingly requires courses to specify in advance the topics of student projects. As discussed above, in the responses to the questionnaire, the principal problems identified by supervisors were related to the complications of making agreements with institutions. To resolve this,

* Universities are under pressure to achieve cost savings in delivery. The supervision of undergraduate projects is expensive when compared with whole class teaching, and coordination with external mentors adds an additional burden. Unless there is a clear institutional policy that authentic experience is essential for students, approaches such as S°C will always be vulnerable to institutional pressure.

4.5.2 Recommendations

If the environment for S°C activities is generally hostile, then it would be wise to abandon the idea that S°C is viable for HE as a whole, and to seek out pockets within the HE system which are more favourable to the S°C approach. There are some universities or institutes that do allow for practical assignments in their curriculum, and it may be that professional education institutes would offer more fertile ground. An alternative niche would be to present S°C as being only for outstanding students, perhaps entitling them to an additional mention in their academic record.

These measures, however, will do nothing to transform the evolution of the curriculum in the European education landscape, and the problems which S°C has encountered with it. We will revisit these wider problems in the following section.

5. Discussion

In the project plan the partners proposed the hypothesis that virtual placements had not been successful because they 'have not to date offered experience of an authentic business environment and business problems. We conclude that for the approach to be successful, these aspects need to be replicated in a virtual placement'. The experience of the project suggests that this hypothesis was incorrect. The question to be addressed is why this was so.

As we have argued in the introductory section, the VALS project was conscientious in carrying out the project plan, and made a large number of presentations to students (more than 600 students have known about the VALS project directly by the presentations made). The fact that the project was so successful in attracting companies and OSS foundations also indicates that the overall performance of the project was not unsatisfactory. No doubt improvements could be made were the project to be repeated, but any shortcomings are not sufficient to explain the lack of take-up among students. Moreover, the decision to develop a virtual placement system (VPS) for VALS meant that the project had complete control over the workflow for virtual placements, and could adjust this as required. The VPS worked well, and all feedback was positive on its functioning. Because of the project's development effort, we can eliminate from our consideration a possible mismatch between the concept of VALS placements and the supporting technology.

One might be led to blame the students for their lack of enthusiasm. However, the students are responding to the lead given by lecturers, who in turn are responding to the institutional pressures. Institutions are in turn responding to a wider policy environment. Project staff who presented to lecturers and students reported that while some universities saw the value in S°C and made an effort to enable the programme despite the difficulties experienced, while other institutions found the challenges created by their culture, systems and processes were too daunting to make the effort. In the light of our experience we are sadly forced to conclude that to develop the S°C beyond a niche role in Europe is not feasible in the current university climate. We now discuss some of the relevant pressures which are being experienced by universities.

5.1 Pressure to provide more content in courses

When universities are requested to show that they meet the needs of industry and society, in many cases their response has been to add curriculum content to courses. In the past, university students had greater freedom to gain experiences both within and beyond the university curriculum, but this is increasingly difficult. The Bologna Process has brought with it many benefits, but has created substantial pressure on curricula (Huisman, Adelman, Hsieh, Shams, & Wilkins, 2012). Firstly, study periods have been shortening in many countries. In Italy, for example, the Bologna process required the education system to change from a single four/five-year degree to a two-tier system characterised by a three-year degree (first cycle, undergraduate) and an additional two-year degree (secondary cycle, master level), in effect shortening the duration of studies required for obtaining a university degree (Cappellari & Lucifora, 2008).

Similar processes have been underway in other European countries, especially Spain and Germany. Secondly the drive for mobility has had impact on curricula. The European Credit Transfer System fosters the intra-European temporary mobility, as students get their studies abroad recognized and credits accumulated abroad are considered for awarding the degree in the home institution in the home country upon return from the mobility (Nicolescu, 2014, p. 233). Nicolescu continues to comment that "the creation of an identical higher education system is ruled out", but be this as it may, the move to exchangeable credits creates pressure on institutions to incorporate new academic content.

As a consequence, universities are bound to a very tight schedule for Bachelor degrees. Nor in many cases is there space in Master degrees for open courses that have a practical perspective, as the validating bodies require them to demonstrate the academic and theoretical status required to this level of study. To facilitate educational processes, programming assignments for students tend to be individual and in a well-defined environment.

The result is that it is hard to fit S°C activities into the curriculum, even though they specifically address the demand from companies for social, communication, collaborative skills and professional problem solving abilities.

5.2 Pressure to provide evidence of efficacy

Many universities across Europe are expected to respond to the implementation of evidence-based educational policies by governments (see Slavin (2002) for an enthusiastic introduction). In these policies the view of 'evidence' which is adopted frequently seems to follow that of William Thompson (Lord Kelvin) in the 19th century, who famously asserted "I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind" (Thompson, 1889). In modern times something similar is asserted by the aphorism "If you can't measure it, you can't manage it.", often attributed (perhaps inaccurately) to Peter Drucker. Following this line of thinking objectives are required to be measurable (Doran, 1981), and more formally so in the Performance Indicator approaches developed by Kaplan and Norton (1992), and applied to education by Fitz-Gibbon (1990) and Barber et al. (2011). There is nothing wrong with seeking to measure things and to interpret the results. However, an insistence that all activities should be measured and managed leads to the twin problems that, on the one hand, learners and lecturers only pay attention to that which will be measured, and, on the other hand, that activities which are not easily measured are ignored or proscribed (see (Seddon, 2008) for a critique of the use of performance indicators in the public sector). The S°C approach requires the university to hand over the activities of the learner to a third party, with no certainty that it will be able to achieve measurable outcomes, and in the knowledge that any measurement cannot be certified by the institution itself. Moreover, if an institution is seeking to maximise the evidence that the services which they provide to students lead to learning outcomes, then they do not benefit from off-loading some of the learning onto a mentor or an OSS community. Within this context, the argument that the experience which students undergo in virtual placements will help them in their future careers is unlikely to meet with a favourable response. The pressure on institutions is to meet performance indicators at the end of the year, and in our experience S°C placements are not perceived to help towards this goal.

Peter Drucker himself is reported to have discussed this point in terms which are very relevant to S°C:

"Your first role . . . is the personal one," Drucker told Bob Buford, a consulting client then running a cable TV business, in 1990. "It is the relationship with people, the development of mutual confidence, the identification of people, the creation of a community. This is something only you can do." Drucker went on: "It cannot be measured or easily defined. But it is not only a key function. It is one only you can perform" (Zak, 2013).

The function which Drucker identifies is the core activity of S°C, and if, as he argues, it cannot be measured or easily defined, then it will always find itself under pressure in an evidence-based policy environment.

5.3 Pressure for increased monitoring and validation of courses and curricula

The increasing monitoring and validation of courses and curricula is in part a response to demands for more content intensive courses, and to the implementation of evidence based policy, and in part a result of the application in the educational context of managerial techniques from industry. One consequence of this trend is that in many cases student project topics have to be specified in the previous year. Such a requirement makes it almost impossible for S°C projects to be included in courses, because the main selling point of S°C is that the projects are authentic tasks which emerge out of the ongoing work of companies and foundations. Consequently, in many cases it is not possible to recognize S°C work as official academic credits under current academic regulations.

5.4 Policy at cross purposes

The pressures which we have identified configure a contradictory environment. The educational reforms which have led to the pressures are intended to increase the relevance of education to industry and society, but the actions which have been taken to achieve this goal have often been counterproductive. The increase in curriculum contents which we have identified is opposed to the idea that students can learn from the experience of a placement. The fiction is maintained that students know what they need to know from course content, and only need a place to practice as an extra on top of that. Thus the practical assignment is conceived as an add on to the curriculum instead of a core part of it. This fiction may be sustainable in a face to face situation, but not in a virtual placement, where the student has nowhere to hide from their ability (or otherwise) to deliver project results. In other words, a face-to-face placement may be conceived of as a matter of having the working experience, of going to an office, being on time, meeting colleagues etc. It is open to the host company to blithely assume the competence of the student to perform professional tasks, or tactfully ignore this on the basis that this competence will be acquired in the first few years of work. In the OSS world, in contrast, success is a matter of achieving a result that is accepted by the community and is according current standards and best practices. However supportive the OSS community may be, the potential puncturing of the fiction of student competence is a substantial threat to students, lecturers and institutions. The failure to confront this problem head-on will, we argue, exacerbate the continued mismatch between higher education and the work environment. Placements, virtual or otherwise, will not make education better if policy is moving in the opposite direction. The university is told to focus on the delivery of course content, while businesses want people who function in the workplace, while policy is deepening the divide. In this context it is open to question if Knowledge Alliances can be an effective response to the problems of education.

It should be recognised that some small but significant success has been achieved using approaches similar to S°C. Demola (http://www.demola.net/) has achieved 350 fulfilled projects, while the Undergraduate Capstone Open-Source Project (UCOSP) project in Canada has over the last four years received more than 300 students from 20 different Canadian universities. However, the Demola initiative developed its strength in Finland, where the education system is quite different from most of Europe, before developing an international presence. Similarly, as Stroulia et al. (2011) explain, most colleges and universities have "open" courses which students can opt to register on. Because of this, mapping UCOSP to a course in a partner institution is not problematic. Such open courses are fast disappearing in Europe. Indeed, university education is increasingly taking on the characteristics of schooling, with the emphasis on completing all the courses in the curriculum in time. For example, Dutch universities which have undergone huge changes since the late nineties under the influence of cuts to the budgets of universities, and to the time and money available to students. Grants were reduced and have recently been transformed into a loan (with interest, but with soft repayment conditions). In this the Netherlands is following the example of similar processes are underway in the UK and other countries, informed by the rationale described in Greenaway and Haynes (2003). In response to these changes, students are focused on completing courses in order to find a job. There are signs that concerns may be emerging about this policy direction, as the Minister of Education in Holland expressed in plans to make higher education more practically relevant, as described in a Dutch article by Benschop (2015).

6. Conclusions

The VALS project brought together many enthusiasts for the use of open source software in education, from both industry and education. The poor response of students to Semester of Code has not dampened this enthusiasm, as is shown by the loyalty of project owners, who were willing to maintain their projects in the Semester of Code for a second round of activity. In education, however, these enthusiasts do not have critical mass in any single institution, and so they do not receive adequate support. Enthusiasts could be more effective in persuading their institutions to engage with OSS and S°C if there were a well-known and established initiative to coordinate the use of OSS in education, whose professionalism was recognised. Unfortunately, no such initiative exists yet. S°C therefore has the opportunity to engage with these people who have commitment to the S°C approach, and to bring them under an umbrella, creating the conditions to move forward together. The Semester of Code Association has been established to take this work forward.

However, the project also provides a valuable perspective on the European education landscape. To this end we have presented our experience of running the S°C initiative, and our theories about the constraints on take up of CoP placements among students. The S°C initiative has not achieved a decisive intervention in the relationship between HE and companies, but this has not been due to lack of effort, or poor processes. Rather the assumptions of the project about the nature of HE, which appeared to be reasonable both to the consortium and to funders, have shown to be false. More specifically, our experience suggests that the preparation which students are given for the move into industry leaves much to be desired, but that this state of affairs is be camouflaged by the qualifications which are awarded, and quality assurance processes which assert that degrees do indeed prepare students for professional life. Some of the challenges we have faced are related to the virtual nature of S°C, for example the lack of confidence among some students in working in English, and the need to coordinate activities remotely. However, we do not believe that these have been the principal barriers. In our experience the main difficulties have been the lack of fit between the rigidity of the university curriculum and the demand for flexibility in industry, and the increasing pressure on universities from government policies. These factors affect face-to-face and virtual placements alike, but in the case of virtual placements many factors which soften the challenge presented to students have been stripped away. Because of this we have argued that virtual placements can be seen as a test of students' ability to undertake authentic tasks. It may be argued that if student placements and Knowledge Alliances are providing authentic professional experience to students, then it is reasonable to expect that this would be sustainable in the context of remote working, which is increasingly common in technical jobs. We have not found this to be the case, and our evaluation suggests that this is, at least in part, because face-to-face placements mask some problems which become evident in virtual placements. Our work with S°C does not constitute an evaluation of the Knowledge Alliance programme, and nor did it set out to throw light on the nature of work placements. But the results which we have obtained should at least lead institutions and policy makers to ask if the benefits of Knowledge Alliances are primarily those of work experience, rather than authentic professional development, and to consider whether they do indeed constitute an effective instrument for the transformation of European higher education.

7. Acknowledgement

With the support of the Lifelong Learning Program of the European Union. Project Reference: 540054-LLP-1-2013-1-ES-ERASMUS-EKA. This project has been funded with support from the European Commission. This publication only reflects the views of the authors, and the Commission cannot be held responsible for any use that may be made of the information contained therein.

References

8. References

Ansell, M. (2016). Snobbery is outdated - universities have to train students for jobs. The Guardian. Retrieved from http://www.theguardian.com/higher-education-network/2016/mar/31/snobbery-is-outdated-universities-have-to-train-students-for-jobs

Barber, M., Kihn, P., & Moffit, A. (2011). Deliverology: From idea to implementation. McKinsey on Government(Spring), 32-39.

Benschop, L. (2015). Bussemaker wil minder "schools" hoger onderwijs. Retrieved from http://www. nu.nl/politiek/4083354/bussemaker-wil-minder-schools-hoger-onderwijs.html

Cappellari, L., & Lucifora, C. (2008). The "Bologna Process" and College Enrolment Decisions (IZA DP No. 3444). Retrieved from Bonn, Germany: http://ftp.iza.org/dp3444.pdf

Casson, A., & Hawthorn, L. (2011). Introducing the Oregon State University Open Source Lab. Technology Innovation Management Review(August 2011), 4-9.

Choy, S. C., & Delahaye, B. L. (2009). University-industry partnership for pedagogy : some principles for practice Proceedings of 16th World Association for Cooperative Education Conference, June 23-26, 2009, Vancouver, Canada.

Fitz-Gibbon, C. T. (Ed.) (1990). Performance indicators. BERA Dialogues. Clevedon, UK: Multilingual Matters Ltd.

García-Peñalvo, F. J. (2011). La Universidad de la próxima década: La Universidad Digital. In C. Suárez-Guerrero & F. J. García-Peñalvo (Eds.), Universidad y Desarrollo Social de la Web (pp. 181-197). Washington DC, USA: Editandum.

García-Peñalvo, F. J. (2015). Entrepreneurial and problem solving skills in software engineers. Journal of Information Technology Research, 8(3), iv-vi.

García-Peñalvo, F. J. (2016). The third mission. Education in the Knowledge Society, 17(1), 7-18. doi:http://dx.doi.org/10.14201/eks2016171718

García-Peñalvo, F. J., Álvarez Navia, I., García-Bermejo, J. R., Conde-González, M., García- Holgado, A., Zangrando, V., Seoane-Pardo, A. M., Cruz-Benito, J., Lee, S., Elferink, R., Veenendaal, E., Zondergeld, S., Griffiths, D., Sharples, P., Sherlock, D., de Toni, A. F., Battistella, C., Tonizza, G., de Zan, G., Papadopoulos, G. A., Kapitsaki, G., Achilleos, A. P., Mettouris, C., Cheung, S., Guerrero, Z., He, E., Alier, M., Mayol, E., Casany, M. J., Wilson, S., Wilson, R., & Johnson, M. (2013). VALS: Virtual Alliances for Learning Society. In F. J. García-Peñalvo, A. García-Holgado, & J. Cruz-Benito (Eds.), Proceedings of the TEEM'13 Track on Knowledge Society Related Projects (pp. 19-26). Salamanca, Spain: Grupo GRIAL.

García-Peñalvo, F. J., & Cruz-Benito, J. (2016). Proyecto Europeo VALS y Semester of Code: Prácticas Virtuales en Empresas y Fundaciones relacionadas con el Software Libre a nivel Europeo. In M. Martín-González (Ed.), Buenas Prácticas sobre la Universidad Digital (pp. 60-67). Madrid: Cátedra UNESCO de Gestión y Política Universitaria.

García-Peñalvo, F. J., Cruz-Benito, J., Conde, M. Á., & Griffiths, D. (2014). Virtual placements for informatics students in open source business across Europe 2014 IEEE Frontiers in Education Conference Proceedings (October 22-25, 2014 Madrid, Spain) (pp. 2551-2555). USA: IEEE. doi:http:// dx.doi.org/10.1109/FIE.2014.7044411

García-Peñalvo, F. J., Cruz-Benito, J., Conde, M. Á., & Griffiths, D. (2015). Semester of Code: Piloting Virtual Placements for Informatics across Europe Proceedings of Global Engineering Education Conference, EDUCON 2015. Tallinn, Estonia, 18-20 March 2015 (pp. 567-576). USA: IEEE. doi:http:// dx.doi.org/10.1109/EDUCON.2015.7096026

García-Peñalvo, F. J., Cruz-Benito, J., Griffiths, D., & Achilleos, A. P. (2015). Tecnología al servicio de un proceso de gestión de prácticas virtuales en empresas: Propuesta y primeros resultados del Semester of Code. IEEE VAEP-RITA, 3(1), 52-59.

García-Peñalvo, F. J., Cruz-Benito, J., Griffiths, D., & Achilleos, A. P. (2016). Virtual placements management process supported by technology: Proposal and firsts results of the Semester of Code. IEEE Revista Iberoamericana de Tecnologías del Aprendizaje (IEEE RITA), 11(1), 47-54. doi:http://dx.doi. org/10.1109/RITA.2016.2518461

García-Peñalvo, F. J., Cruz-Benito, J., Griffiths, D., Sharples, P., Willson, S., Johnson, M., Papadopoulos, G. A., Achilleos, A. P., Alier, M., Galanis, N., Conde, M. Á., Pessot, E., Elferink, R., Veenendaal, E., & Lee, S. (2014). Developing Win-Win Solutions for Virtual Placements in Informatics: The VALS Case. In F. J. García-Peñalvo (Ed.), Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'14) (pp. 733-738). New York, USA: ACM. doi:http:// dx.doi.org/10.1145/2669711.2669982

García-Peñalvo, F. J., Griffiths, D., Cruz-Benito, J., Veenendaal, E., Achilleos, A., Wilson, S. & Kapitsaki, G. (2015). Report on virtual placements pilots and evaluation of their results - Virtual Alliances for Learning Society (VALS) project & Semester of Code. Salamanca, Spain: Virtual Alliances for Learning Society Project Consortium. University of Salamanca, University of Bolton, Raycomm B.V., University of Cyprus, University of Oxford. doi:10.6084/m9.figshare.3198691

Greenaway, D., & Haynes, M. (2003). Funding higher education in the uk: the role of fees and loans. The Economic Journal, 113(485), F150-F166. doi:http://dx.doi.org/10.1111/1468-0297.00102

Huisman, J., Adelman, C., Hsieh, C. C., Shams, F., & Wilkins, S. (2012). Europe's Bologna process and its impact on global higher education. In D. K. Deardorff, H. de Wit, J. D. Heyl, & T. Adams (Eds.), The SAGE Handbook of International Higher Education (pp. 81-100). Thousand Oaks: Sage Publications. doi:http://dx.doi.org/10.4135/9781452218397.n5

Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard - Measures that drive performance. Harvard Business Review, 70(1), 71-79.

Missett, T. C., & Foster, L. H. (2015). Searching for Evidence-Based Practice: A Survey of Empirical Studies on Curricular Interventions Measuring and Reporting Fidelity of Implementation Published During 2004-2013. Journal of Advanced Academics, 26(2), 96-111. doi:http://dx.doi. org/10.1177/1932202X15577206

NCSU Open Source Initiative. (2014). Open Source NC State University. Retrieved from https:// opensource.ncsu.edu/

Nicolescu, L. (2014). Convergence and Internationalization of Higher Education, The Case of Romania. In A. M. Dima (Ed.), Handbook of Research on Trends in European Higher Education Convergence (pp. 219-241). Hershey PA: Information Science Reference.

Seddon, J. (2008). Systems Thinking in the Public Sector: The Failure of the Reform Regime and a Manifesto for a Better Way. Axminster: Triarchy Press.

Slavin, R. E. (2002). Evidence-Based Education Policies: Transforming Educational Practice and Research. Educational Researcher, 31(7), 15-21. doi:http://dx.doi.org/10.3102/0013189X031007015

Stroulia, E., Bauer, K., Craig, M., Reid, K., & Wilson, G. (2011). Teaching Distributed Software Engineering with UCOSP: The Undergraduate Capstone Open-Source Project. Proceedings of the 2011 Community Building Workshop on Collaborative Teaching of Globally Distributed Software Development - CTGDSD '11 (pp. 20-25). New York, NY, USA: ACM. doi:10.1145/1984665.1984670

Taylor, S. (2016, March 9). Goodnight Melange. Retrieved from http://google-opensource.blogspot. com.es/2016/03/goodnight-melange.html doi:http://dx.doi.org/10.1145/1984665.1984670

Thompson, W. (1889). Electrical Units of Measurement. Popular Lectures and Addresses (Vol. 1, pp. 73-136). London: Macmillan & Co.

Zak, P. (2013). Measurement myopia. Retrieved from http://www.druckerinstitute.com/2013/07/ measurement-myopia/