Digital Maturity Framework for Higher Education Institutions

Abstract

Digital technologies in educational institutions have the potential to be one of the main catalysts of quality education. The purpose of this paper is to analyse digital maturity frameworks, to describe their development methodologies and to propose a Digital Maturity Framework for Higher Education Institutions (DMFHEI). The methodology for framework development consists of five main steps: literature search and data extraction; two focus group analyses; applying the Q-sorting method; reviews by two experts; and a summary of results. Results presented in this paper show that the DMFHEI has recognised seven areas that can be divided into 43 describable elements.

Headnote

Abstract. Digital technologies in educational institutions have the potential to be one of the main catalysts of quality education.

The purpose of this paper is to analyse digital maturity frameworks, to describe their development methodologies and to propose a Digital Maturity Framework for Higher Education Institutions (DMFHEI). The methodology for framework development consists of five main steps: literature search and data extraction; two focus group analyses; applying the Q-sorting method; reviews by two experts; and a summary of results. Results presented in this paper show that the DMFHEI has recognised seven areas that can be divided into 43 describable elements.

Keywords. digital maturity, framework, higher education, higher education institutions

1Introduction

Since the end of the 20th century, developed countries have realised the importance of establishing a strategy for developing and integrating new technologies in all areas of civil and economic life. In this context, ereadiness can be defined as the degree to which a community is prepared to participate in the networked world (CID, 2000) and the degree to which an economy or community is prepared to participate in the digital economy (APEC, 2000).

According to the European Commission's report (2013) 'Opening up education through new technologies', rapid digital change in our society and economy mean greater demand for digital skills and competences. Education and training must address this need, which requires investment in infrastructure (e.g., broadband, digital devices), teachers' training, organisational change and the development of high quality educational resources, including apps and software. Technology, when used correctly, can also help us to learn better, more efficiently and creatively and make it possible to access wider and more up-todate sources of knowledge.

Digital technologies enable change in learning and teaching, but these changes do not guarantee sustainability. Changes require a multiple-system approach, including investing in infrastructure, investing in the professional development of professors, changing curricula, reviewing how students are assessed and professors are evaluated, promoting and encouraging collaboration and open content, integrating everything into a well-run environment and quality control.

Digital technologies in educational institutions have the potential to be one of the main means of delivering quality education in line with their mission and vision. For this reason, the adoption of digital technologies and their integration into educational systems is considered a form of educational innovation and implies changes in three basic dimensions: pedagogical, technological and organisational.

Using a digital maturity framework and instrument for higher education institutions (HEI) makes it possible to estimate the maturity level at which an HEI is currently estimated, to identify all needed improvement areas, to identify which recommendations need to be made and to specify in which areas and elements the level of digital maturity should be raised. An e-readiness assessment allows for the positioning, or determining of the starting point, of an institution's readiness for participation in the information society.

E-readiness is the degree to which an HEI is prepared and ready to participate in the adoption of ICT. The readiness of an HEI to adopt digital technologies is an important prerequisite to becoming a digitally mature institution.

With the above in mind, digital maturity frameworks and instruments for HEIs contain identified digital maturity areas and elements that allow for an initial point assessment of an HEI, which is included in the e-readiness concept, and includes the ability to provide recommendations for improvements, such as an upgrading in relation to the concept of ereadiness.

This paper is divided into the following sections: an introduction, presented in Section 1; a systematic literature analysis of the digital maturity frameworks used in this study, and an explanation of the research methodology in Section 2; a focus on summarising research results and presenting the DMFHEI, in Section 3; and a conclusion, presented in Section 4.

2Research Methodology

The development of a digital maturity framework aims to identify areas and elements of HEIs' digital maturity and to identify key areas and elements needed to raise their levels of digital maturity. Developing the framework requires the application of a complex methodology, which includes a set of methods, techniques and instruments, such as qualitative analysis and the comparison of similar frameworks that describe digitally mature organisations from the perspective of the concept and strategic documents at the national and international level, analyses of existing projects' documentation, the Q-sorting method and focus groups.

The developed framework has areas and elements that are not mutually exclusive or disjointed. Moreover, they are complementary and interconnected, thus forming a unified whole. The framework development methodology, as well as the areas and elements of the framework that are important for determining an HEI's level of digital maturity, are described in detail in Section 3.

A qualitative analysis of the literature identified 15 digital maturity frameworks, which are further analysed and presented in this section. They are as follows: 1) Assessing the e-Maturity of your School (Ae-MoYS); 2) DigCompOrg (DigCompOrg); 3) eLearning Roadmap (eLearning Roadmap); 4) eLemer (eLemer); 5) The ePortfolios & Open Badges Maturity Matrix (ePOBMM); 6) Future Classroom Maturity Model (FCMM); 7) HEInnovative (HEInnovative); 8) Jisc Strategic ICT Toolkit (JISC); 9) Ledning, Infrastruktur, Kompetens, Användning (LIKA); 10) Microsoft Innovation Framework & Self-reflection Tool (Microsoft IF & SRT); 11) NACCE SRF (NACCE SRF); 12) OPEKA (OPEKA); 13) Upscaling Creative Classrooms in Europe (SCALE CCR); 14) School Mentor (School Mentor); and 15) Venstress (Venstress).

Within the present analysis, special attention was paid to the following elements: the existence of an accompanying instrument; the framework's areas, elements and descriptors; implemented development approaches; application areas and examples of their best practices. Below is a brief description of each analysed framework.

1. Ae-MoYS is a framework and online selfevaluation questionnaire described by 5 areas and 30 descriptors. It employs both qualitative and quantitative development approaches with application areas at the elementary and high school levels. Its best practice examples are in the EU.

2. DigCompOrg is a framework described by 5 areas, 15 elements and 74 descriptors. It uses a qualitative development approach with application areas at the elementary school, high school and HEI levels and best practice examples throughout the world.

3. eLearning Roadmap is a framework and matrix described by 5 areas and 108 descriptors. It uses a qualitative development approach with application areas in elementary and high schools and best practice examples in Ireland.

4. eLEMER is a framework and online self-evaluation questionnaire described by 4 areas, 40 elements and 10 descriptors. It makes use of both qualitative and quantitative development approaches with application areas in elementary and high schools and best practice examples in Hungary.

5. ePOBMM is a framework and matrix described by 7 areas and 300 descriptors. It employs a qualitative development approach with application areas mostly in HEIs and best practice examples in the EU.

6. FCMM is a framework and online self-evaluation questionnaire described by 5 areas and 25 descriptors. It uses a qualitative development approach with application areas in elementary and high schools and best practice examples in the EU.

7. HEInnovative is a framework and online selfevaluation questionnaire described by 7 areas and 44 descriptors. It takes a qualitative development approach with application areas in HEIs and best practice examples throughout the world.

8. JISC is a framework and online self-evaluation questionnaire described by 6 areas and 69 descriptors. It uses both qualitative and quantitative development approaches with application areas in HEIs and best practice examples in the EU.

9. LIKA is a framework and online self-evaluation questionnaire described by 4 areas and 78 descriptors. It adopts a qualitative development approach with application areas in elementary and high schools and best practice examples in Sweden.

10. Microsoft IF & SRT is a framework and online selfevaluation questionnaire described by 4 areas, 16 elements and 96 descriptors. It uses both qualitative and quantitative development approaches with application areas in elementary and high schools and best practice examples from around the world.

11. NACCE SRF is a framework and online selfevaluation questionnaire described by 6 areas, 11 elements and 220 descriptors. It makes use of both qualitative and quantitative development approaches with application areas at the nursery, elementary and high school levels and best practice examples in the United Kingdom.

12. OPEKA is a framework and online self-evaluation questionnaire described by 3 areas, 17 elements and 145 descriptors. It uses both qualitative and quantitative development approaches with application areas in elementary and high schools and best practice examples in Finland.

13. Scale CCR is a framework described by 8 areas and 28 elements. It employs a qualitative development approach with application areas in elementary and high schools and best practice examples from throughout Europe.

14. School Mentor is a framework and online selfevaluation questionnaire described by 6 areas and 150 descriptors. It makes use of both qualitative and quantitative development approaches with application areas in elementary and high schools and best practice examples in Norway.

15. Venstress is an online self-evaluation questionnaire described by 20 indicators. It takes a qualitative development approach with application areas in elementary and high schools and best practice examples in the Netherlands.

Our analysis revealed that DigCompOrg, on account of its favourable characteristics, is best suited to serve as the basis for creating the Digital Maturity Framework for Higher Education Institutions. It is worth mentioning that further modifications and adjustment are needed because the mentioned framework needs to be adjusted for HEIs. DigCompOrg provides a comprehensive and generic conceptual framework that reflects on all aspects of the process of systematic integration of digital learning into an educational institution. The framework is primarily designed to focus on learning and teaching and the activities undertaken by the educational institution to support learning. DigCompOrg includes domains, subdomains and descriptors that can be considered organisational (e.g., ICT infrastructure) or individual responsibility (e.g., learning and teaching).

DigCompOrg is a framework for digitally competent educational institutions and includes all the main areas of digitally competent educational institutions but its main purpose is to serve as a generic framework. It represents a very complex and comprehensive framework that can be the basis for development of specific framework and assessment tool of specific educational systems. So we have used DigCompOrg as a base in developing our Framework.

A digitally competent educational institution needs a balanced mix of strong leadership and management, staff and stakeholders who are willing to take personal responsibility for independently running actions and initiatives leading to the prosperity of the educational institution.

The initial set of the DMFHEI are defined (see Table 1) based on the results of the qualitative analysis of the e-readiness assessment tools and qualitative analysis of the digital maturity frameworks.

The results presented in Table 1 were generated using the methodology we have described (areas and elements defined based on qualitative analyses).

The results of applying the entire method are shown in Section 3.

The Digital Maturity Framework for Higher Education Institutions (DMFHEI) has been in development since July 2015. The methodological approach we used for the process was mostly qualitative. In the first phase, we completed a qualitative analysis of 15 frameworks for digital maturity, with a particular focus on information and communication technologies (ICT), as explained in Section 2.

After conducting qualitative analyses of the ereadiness assessment tools and of the digital maturity assessment frameworks, in the second phase of the framework development process, two focus group studies were held to obtain input from experts on defining new framework areas and their elements. The focus group method is a qualitative form of research involving a group discussion about a given topic. The primary focus of such a group is to initiate a deep discussion and to explore the values or attitudes of respondents to a problem or topic.

The first focus group was heterogeneous (comprised of university professors, PhD students, a representative of the University Computing Centre - SRCE and a representative of the IGH institute) with 15 participants (N=15), of whom four were PhD students (N=4). The second focus group was also heterogeneous (university professors and PhD students) and had ten participants (N=10) three of which were PhD students (N=3). The results of the qualitative analyses of 15 frameworks for digital maturity, and the results of the two focus groups, led to the proposal of seven areas (leadership, planning and management; quality assurance; scientific-research work; technology transfer and service to society; learning and teaching; ICT culture; and ICT resources and infrastructure) and 53 elements for the DMFHEI.

In the second phase of the framework development process, the sorting cards (Q-sorting) method was applied. The Q-sorting method is a theoretically based quantitative tool for examining opinions and attitudes. The method enables researchers to examine human subjectivity systematically and quantitatively. Participants are experts who are theoretically relevant to the research problem. During the Q-sorting process, experts were asked to sort 53 element cards into seven proposed areas. After this was complete, we calculated a content validity ratio (CVR).

The CVR calculation formula was proposed by Lawshe (1975); according to it, only items that more than 50% of respondents considered to be valid / understandable were retained in further research steps.

The CVR formula is:

CVR=(n - N / 2) / (N/ 2).(1)

In Lawshe's formula, n denotes the number of participants that a particular variable (in this case, an element) considers desirable. N represents the total number of participants in the Q-sorting method. When analysing data, each of the elements is described on a three-step scale: 1 - valid/understandable, 2 - desirable, 3 - non-relevant.

The CVR was calculated for each of the 53 proposed elements.

After the Q-sorting and calculation of CVR were complete, the number of elements decreased by ten, from 53 to 43 (Table 2). The ten elements not recognised as important during Q-sorting were: administrative support for ICT; employee education on ICT application; employees' informal learning; students' experience with ICT applications; HEIs' intellectual property licensing; enlightenment and participation of employees in training programmes for the development of digital competences; planning and procurement of ICT infrastructure; access to ICT research infrastructures; regulated access to ICT resources; and wider digital environment (monitoring global trends in HEIs). The other 43 elements are presented in Section 3.

In the third development phase, the results obtained by qualitative analysis, the research from the two focus groups, the Q-sorting process and the calculation of CVR ratio were analysed by two external experts to yield a proposal for the DMFHEI.

3Digital Maturity Framework for HEI

The proposed Digital Maturity Framework for Higher Education Institutions (DMFHEI) consists of seven areas, which are presented in Figure 1.

Each area consists of a larger number of elements. Due to length limitations, the other six areas (quality assurance; scientific-research work; technology transfer and service to society; learning and teaching; ICT culture and ICT resources and infrastructure) are not shown in figure form.

The quality assurance area consists of six elements: ICT quality assurance policies; monitoring and periodic review of study programmes, from the aspect of ICT application; evaluation of the work of teaching, research, administrative and technical staff; continuous monitoring of the results of scientific-teaching work and progress; procedures for determining the needs, development or acquisition of ICT resources and their application; approved procedures and follow-up on student enrolment, their progress through study and the completion of studies supported by ICT.

The scientific-research work area consists of six elements: the use of ICT in the preparation and publication of scientific papers; ICT support in the preparation and management of scientific research work and projects; ICT research (collaborative ICT research on HEIs); a system of support for researchers at the beginning of their careers in applying ICT in scientific research; continuous training of researchers in ICT application in scientific research; and networking and collaboration of researchers with ICT support.

The technology transfer and service to society area consists of three elements: collaboration with stakeholders (employers, local community, pre-tertiary education) supported by ICT; applied research and professional projects supported by ICT and/or ICT; and networking of researchers and users of research (stakeholders) supported by ICT.

The learning and teaching area consists of seven elements: preparation, storage and use of digital content in learning and teaching; innovative learning and teaching methods with ICT; the development of teachers' digital competence; the development of students' digital competence; the use of learning analytics to improve learning and teaching; ubiquitous learning and open curricula; and personalisation and support for under-represented groups by using ICT in learning and teaching.

The ICT culture area consists of six elements: the network presence of HEIs; using ICT in HEI promotion; the development of digital literacy and the promotion of innovativeness in ICT application with HEI employees; self-confidence and motivation of employees on the importance of ICT application; providing access to and support in the application of ICT infrastructure; and the application of ethical standards, copyrights and intellectual property in the ICT field.

The ICT resources and infrastructure area consists of seven elements: the availability of ICT resources (hardware and software) for learning and teaching; the availability of ICT resources for scientific research; network infrastructures at HEIs; access to ICT resources for students (both in and out of the classroom); digital environment and information systems for employees and students; technical support and maintenance of ICT resources at HEIs; and information security system.

After determining the DMFHEI area and elements, the next step was to determine the descriptions of each element. Due to space limitations, we are not able to show descriptors for all 43 elements.

4 Conclusion

The purpose of this paper is to analyse digital maturity frameworks, describe development methodology and propose the Digital Maturity Framework for Higher Education Institutions (DMFHEI). The DMFHEI identifies seven areas (leadership, planning and management; quality assurance; scientific-research work; technology transfer and service to society; learning and teaching; ICT culture; and ICT resources and infrastructure) with 43 elements.

One limitation of research is the number of domain elements based on which a measuring instrument for digital maturity of higher education will be developed. The instrument will require a high level of concentration and time resources of stakeholders involved in the research.

The next step in this research is creation of the instrument by organising areas as a rubric with elements. Each element should be described on five levels of maturity. Testing and feedback of a rubric will be conducted through a series of expert consultation, workshops, focus groups and semi-structured interviews.

After completion, developed instrument (rubric based on the DMFHEI) can be used as a tool in the form of a self-evaluation questionnaire to evaluate an HEI's digital maturity level and to identify the areas that need to be improved for the better prosperity of the institution relative to ICT development.

Pilot testing of developed instrument will be conducted at several higher education institutions.

Acknowledgments

The Croatian Science Foundation supported this work under project IP-2014-09-7854.

Footnote

1 CVR calculation formula proposed by Lawshe

References

References

Ae-MoYS. (2011). Assessing the e-maturity of your school. Retrieved from http://e-mature.ea.gr/

APEC readiness initiative 2000. (2000). E-commerce readiness assessment guide. Retrieved from http://publications.apec.org/publicationdetail.php?pub_id=647

Brown, S. R. (1980). Political subjectivity: Applications of Q methodology in political science. New Haven, CT: Yale University Press.

Chen, I., Kidd, TT. (2011). Ubiquitous Learning : Strategies for Pedagogy, Course Design, and Technology. Charlotte, NC : Information Age Publishing.

Choucri, N., Maugis, V., Madnick, S., & Siegel, M. (2003). Global e-readiness - for what? Cambridge: Massachusetts Institute of Technology.

CID. (2006). Readiness for the networked world - A guide for developing countries. Retrieved from http://cyber.law. harvard. edu/readinessguide/guide. pdf

DigCompOrg. (2015). Digitally competent educational organisations. Retrieved from https://ec.europa.eu/jrc/en/digcomporg/framework

E-learning Roadmap. (2009). Retrieved from http://www.ncte.ie/elearningplan/roadmap/

eLEMER. (2010). ICT and School Development. Retrieved from <http://ikt.ofi.hu/english/>

ePOBMM. (2013). The ePortfolios & Open Badges Maturity Matrix. Retrieved from http://www.eportfolio.eu/matrix

European Association for Quality Assurance in Higher Education (ENQA). (2015). The Standards and guidelines for quality assurance in the European Higher Education Area (ESG). Retrieved from http://www.enqa.eu/index.php/home/esg/

European University Association. (2015). Trends 2015: Learning and Teaching in European Universities. Retrieved from http://www.eua.be/policy-representation/highereducation-policies

European Commission. (2013). Opening up education through new technologies. Retrieved from https://ec.europa.eu/education/policy/strategicframework/education-technology_en

FCMM. (2010). Future classroom maturity model. Retrieved from http://fcl.eun.org/hr/toolset2

Finne, H. et al.(2011). Report from the European Commission 's Expert Group on Knowledge Transfer Indicators. Retrieved from <http ://ec.europa. eu/research/innovationunion/pdf/kti-report-final.pdf>

HEInnovative. (2013). HEInnovate. Retrieved from https://heinnovate.eu/

Higher Education Funding Council for England (HEFCE). (2016). Report to the UK higher education sector and HEFCEs. Retrieved from <http ://www. hefce.ac .uk/pubs/rereports/year/2016 /ketech/>

JISC. (2010). Jisc strategic ICT toolkit. Retrieved from https://www.jisc.ac.uk/guides/managingcourse-informationAe-MoYS.

Kampylis P., Punie Y., & Devine J. (2015). Promoting effective digital-age learning. European Union: European Union Joint Research Centre.

Lawshe C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575.

LIKA. (2013). Ledning, infrastruktur, kompetens, användning. Retrieved from http://www.iktpedagogerna.se/lika-it-tempen-paskolan/

Microsoft IF & SRT. (2009). Microsoft Innovation framework & self-reflection tool. Retrieved from http://www.is-toolkit.com/self_reflection.html

NACCE SRF. (2005). Self-review Framework (SRF). Retrieved from https://www.naacesrf.com/

National Research Council: Improving Measurement of Productivity in Higher Education. (2012). Retrieved from <http://biblioteca.ucv.cl/site/colecciones/manuales _u/13417.pdf>

OPEKA. (2012). Opeka - System Design. Retrieved from http://opeka.fi/Opeka-SystemDesign-1.0.pdf

Pažur Aničić, K., Dívjak, B., Arbanas, K. (2016). Preparing ICT Graduates for Real-World Challenges: Results of a Meta-Analysis. IEEE Transactions on Education 60 (3), 191-197.

Scale CCR. (2012). Up-scaling creative classrooms in Europe. Retrieved from http://is.jrc.ec.europa.eu/pages/EAP/SCALECCR. html

School Mentor. (2014). The Norwegian centre for ICT in education. Retrieved from https://iktsenteret.no/sites/iktsenteret.no/files/ikt_ ministerbrosjyre_eng.pdf

Venstress. (2008). Scholen op de kaart. Retrieved from https ://www.scholenopdekaart.nl/

Zoukkari, N. B., Doukkali, A., & Regragui, B. (2012). E-strategy formulation: A new approach based on a layered model. Journal of Theoretical and Applied Information Technology, 36(1), 101-112.