E-learning educational atmosphere measure (EEAM): a new instrument for assessing e-students’ perception of educational environment

Abstract

Universities assess their academic learning environment to improve students’ learning. Students’ experience in e-learning environment is different from face-to-face educational environment. So, in this study a specific valid and reliable instrument was devised for assessing perception of e-students from educational environment, that is, educational atmosphere. Firstly, we devised the primary instrument based on factors constituting educational atmosphere. Then Instrument’s content and construct validity were assessed. Also, Cronbach’s alpha and test–retest were used for studying the internal consistency and reliability of the instrument respectively. The final instrument named ‘e-learning educational atmosphere measure’ (EEAM) consisted of 40 items covering six factors, including programme effectiveness, teaching quality, ethics and professionalism, learner support, safety and convenience, and awareness of the rules, which accounted for 68.53% of variances. Content validity ratio was more than 0.51 and content validity index score of all questions was above 0.81. Test–retest reliability was 0.85 (p = 0.001) and Cronbach’s alpha was 0.943. Assessing educational atmosphere in e-learning settings by EEAM could provide managers and investors with useful information to settle an effective education system by prioritising the necessary changes.

Introduction

E-learning environment is a web based system that provides the possibility of conducting teaching–learning processes using software tools and applications (Cassidy 2016). Being student-centred, self-directed, interactive, flexible and self-paced are considered as advantages of this environment that could lead to persistent and effective learning (Hampel and Dancsházy 2014). In fact, this new learning environment is based on information and communication technology (ICT), in which a wide range of asynchronous and synchronous tools are used (Bdiwi et al. 2019). These tools provide the opportunity for students and teachers interactions; meanwhile, as there is no face-to-face classroom sessions, students need different communication tools (Hung et al. 2010). This makes e-learning environment and consequently students’ experience different from face-to-face environment. Learning environment is one of the factors affecting students’ behaviour and educational atmosphere is students’ perception of the environment (Genn 2001). Students’ experience of educational atmosphere may be related to factors such as their academic development, satisfaction and achievement (Genn 2001; Shirazi et al. 2014). Besides, recent studies have demonstrated that learning environments, into which advanced technologies are integrated, facilitate acquiring knowledge, skill and attitude (Chang et al. 2015). Different factors, namely university, student, admission, registration and environmental or physical elements constitute educational atmosphere (Genn 2001), although factors creating educational atmosphere specifically in an e-learning setting were extracted in a study published in 2017 (Mojtahedzadeh 2017).

On the other hand, as quality has been noticeably considered in universities and institutes as a strategy for competing with their peers, they assess their academic learning environment to promote innovation and enhance curriculum quality. This shows the importance of assessing educational atmosphere for its improvement. Meanwhile considering the differences between virtual and face-to-face environments, a specific instrument is needed for the assessment of educational atmosphere in e-learning setting (Lee and Lee 2008). Reviewing previous studies showed that most of them focused on evaluating educational atmosphere whether in settings based on technology features or pedagogical approaches of their own time or just some components of blended or distant learning environments rather than e-learning setting (Aldridge, Dorman, and Fraser 2004; Jegede, Fraser, and Fisher 1995; Trinidad, Aldridge, and Fraser 2005; Walker and Fraser 2005).

Theoretical background

In order to build up the theoretical frame work of the research, we reviewed related studies that had defined factors influencing some aspects of educational environment in e-learning. For this purpose, we performed an advanced search between 2000 and 2018 in the Web of Science and SCOPUS databases with the following keywords of survey: questionnaire, instrument or factors in combination with online learning, distance learning, e-learning or virtual learning and learning environment or educational atmosphere. We selected relevant articles and read them thoroughly. The most relevant articles, from the oldest to the recent ones, were as follows:

One of the first related studies was by Taylor and Maor (2000), in which the constructivist online learning environment survey (COLLES) was designed to assess students’ preferred online learning environment in comparison with their actual experiences. In their e-learning context, computer-assisted communication tools were used to exchange learning materials and to interact with peers and instructors. They focused on courses designed regarding constructivist approach. It evaluated students’ and tutors’ perception of professional relevance, reflective thinking, interactivity, cognitive demand, affective support and interpretation of meaning.

Another study was by Chang and Fisher (2001), which focused on web-based learning and developed an instrument called ‘Web-based Learning Environment Instrument’ (WEBLEI) to assess students’ perception of such learning. They considered four scales: emancipatory activities (including activities’ convenience, efficiency and autonomy), co-participatory activities (including flexibility, reflection, quality, interaction, feedback and collaboration), qualia (including enjoyment, confidence, accomplishments, success, frustration and tedium) and information structure and design activities (including content relevance, content validity, accuracy and balance, navigation, and affective aspects). In this study, the learning setting was a blended one comprising face-to-face classes, e-mail, phone, bulletin board, chat line, online materials and remote library.

Thereafter, Aldridge, , Dorman, and Fraser (2004) validated an instrument named as ‘Technology-Rich Outcomes-Focused Learning Environment Inventory’ (TROFLEI) to assess new classroom environments of the time. TROFLEI consisted of 80 items assessing 10 dimensions in high school classroom environments: student cohesiveness, teacher support, involvement, investigation, task orientation, cooperation, equity, differentiation, computer usage and young adult ethos. By technology, authors meant using computers for learning activities (e.g. email, the Internet and forums), and not a complete e-learning system as a whole.

Another instrument, Online Learning Environment Survey (OLES), was developed by Trinidad, Aldridge, and Fraser (2005) to assess students’ perceptions of e-learning part of their blended classes. In this study, e-learning was defined as ‘technology enhanced learning and/or technology delivered learning’. OLES consisted of seven scales: computer usage, teacher support, student interaction and collaboration, personal relevance, authentic learning, student autonomy, and asynchronicity.

‘Distance Education Learning Environments Survey’ (DELES) was another tool developed by Walker and Fraser (2005). DELES had six scales: instructor support, student interaction and collaboration, personal relevance, authentic learning, active learning, and student autonomy. As the instrument’s name shows, it focused on distant environments that were not necessarily enriched with e-learning facilities. Nevertheless, no precise description was provided for distant courses targeted in this study.

Also, Clayton (2007) developed a measure, ‘Online Learning Environment Survey’ (OLLES), which aimed at providing educationalists with information on the delivered online learning environments from students’ perspective. Instrument’s scales were computer competence, material environment, student collaboration, tutor support, active learning, information design and appeal, and reflective thinking. In this study, participants were university students that the tutors were employing some kind of online learning in their traditional courses, that is, blended learning model.

Here we reach a newer study in which authors conducted semi-structured in-depth interviews with e-teachers and e-students to determine perceived factors influencing educational atmosphere in e-learning environment. For devising interview’s framework, they had used previous literature. Participants were selected purposefully among students and faculty members who were involved in PhD and MSc virtual programmes. Courses were delivered via a learning management system (LMS). Interviews’ content and analysis led to six themes, including status of virtual education, learner support (including systemic, cognitive and emotional ones), teaching skills (including cyberspace teaching and content preparing skills), evaluation (including students’ evaluation methods and evaluation of educational system by students), professionalism and professional ethics (respecting intellectual property rights, and observing cultural issues and etiquette norms), and self-efficacy (strengthening problem-solving, learning and meta skills). These themes were considered as factors that comprised educational atmosphere in e-learning setting (Mojtahedzadeh 2017).

Reviewing the literature, we couldn’t find any study introducing an instrument for the assessment of educational atmosphere in current e-learning environments from e-students’ perspective. So, this study aims at designing a valid and reliable tool for assessing educational atmosphere in e-learning setting based on factors creating such an environment.

Method

We considered the above-mentioned study in which factors creating educational atmosphere in an e-learning setting were determined (Mojtahedzadeh 2017) as the theoretical framework of this applied research. The reasons for selecting this study were as follows:

• Considering the time of the study, it was in accordance with current e-learning pedagogical approach and technology.

• The study’s participants were faculty members and students of virtual programmes, not the blended ones, so that they had the opportunity of attaining deeper understanding of e-learning environment.

• Viewpoints of both faculty members and students were explored for determining the influencing factors on educational environment.

Thereafter, in this study, we devised the items of the instrument based on factors explored in the above-mentioned article and assessed its psychometric properties.

We thoroughly presented factors creating educational atmosphere in an e-learning setting to a committee comprising five e-learning and education experts who devised the draft of an instrument for its assessment. The draft was reviewed using the comments of other 10 experts in e-learning and education to assess its face validity. Besides, 10 e-learning and education experts evaluated the instrument’s content validity by determining coefficients of content validity ratio (CVR) and content validity index (CVI). For CVR, experts were asked to review each item based on a three-part spectrum of necessary, useful but not necessary and not necessary, and CVI was evaluated based on a four-part spectrum of irrelevant, somehow relevant, relevant, but needs to be revised and fully relevant. The instrument was then revised based on the results.

In order to check the internal consistency of the instrument, Cronbach’s alpha was used. For this purpose, the instrument was presented to 185 students studying virtual programmes by e-mail or in-person. These were master programmes of ‘e-Learning Planning’, ‘Medical Education’, ‘Educational Technology’, ‘Library and Information System’, and ‘Family Medicine Management’. Moreover, for measuring test–retest reliability, 25 participants from the original sample group re-filled the instrument after 4 weeks.

To determine the construct validity of the questionnaire, the Principle Component Analysis (PCA) method was utilised. To examine the feasibility of this analysis, the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and the Bartlett’s test of sphericity were administered. In order to determine the number of questionnaire factors, Eigen values indices, the fraction of total variance explained by each factor and the scree plot were investigated. To simplify the expression of data, the Varimax rotation method was used.

This study was conducted in accordance with research ethical standards and its ethical approval was obtained from the university’s ethics committee.

All quantitative statistics were analysed using SPSS (IBM SPSS statistics for windows, Version 21.0. Armonk, NY: IBM Corp.).

Results

The expert committee devised 43 items as instruments’ questions that they believed would cover assessing educational atmosphere in an e-learning environment. All the questions were designed in the 5-point Likert scale, including totally agree, agree, neutral, disagree and absolutely disagree (rated 5 to 1).

Other 10 experts approved instrument’s face validity. Regarding content validity, the CVR value was greater than 0.51, and the CVI score of all questions was above 0.81. Thus, content validity of the scale was also confirmed (Ayre and Scally 2014).

From 185 virtual students to whom the instrument was administered, 138 participants (response rate: 74.5%) completed the questionnaire. These students were studying virtual master’s programmes of ‘e-Learning Planning’ (32 students), ‘Medical Education’ (24 students), ‘Educational Technology’ (20 students), ‘Library and Information System’ (18 students), and ‘Family Medicine Management’ (44 students). The test–retest reliability of the instrument was 0.85 (p = 0.001) and the Cronbach’s alpha for the whole instruments was 0.943.

Regarding PCA, the KMO index was 0.812, indicating the adequacy of the sample size. The Bartlett’s test also showed the suitability of the data for factor analysis (p = 0.001). In another factor analysis, 10 factors were extracted at first. The lowest acceptable factor load was considered at 0.35 (Dadgaran et al. 2016) so that factors with lesser amounts were eliminated. After reviewing various multi-factor solutions and considering the content and subject areas of items, finally the six-factor solution was selected. Three items that had no effect on the total explained variance and were not loaded in any of the six principle factors were omitted (number 36, 38 and 39).The final instrument, named as ‘e-learning Educational Atmosphere Measure’ (EEAM), included 40 questions that covered six factors consisting of programme effectiveness, teaching quality, ethics and professionalism, learner support, safety and convenience, and awareness of the rules. The scree value of each of the six factors was greater than one and these six factors accounted for 68.53% of the total variance of the variables. Table 1 shows rotated matrix of loaded items in these six factors. Also, the rotated graph of the scree values is demonstrated in Figure 1. Finally, Table 2 shows the EEAM’s factors and items.

Discussion

The aim of this study was to design an instrument for assessing educational atmosphere, that is, student’s perception of educational environment, in an e-learning setting. For this purpose, the draft of the instrument was devised based on the theoretical framework, and after psychometric analysis, EEAM comprising 40 five-point Likert-type items was finalised. It covered six factors, including programme effectiveness, teaching quality, ethics and professionalism, learner support, safety and convenience, and awareness of the rules, which were considered as perceptive factors for a student to understand educational environment.

The first known factor in this study is the ‘programme effectiveness’, which includes such issues as what learners have learned during the course, preparing them for their future career, the possibility to learn academic-related skills, the increased ability to interact with others in cyberspace, resources’ attractiveness, clear assignments and contents, proper assessments as well as students’ satisfaction. In Chang and Fisher’s (2001) instrument, ‘web-based learning environment inventor’, the scales of ‘emancipatory activities’ and ‘information structure and design activities’ comprised the factors of ‘convenience, efficiency and autonomy’, and ‘appropriate content and material design’ respectively. Also, in the questionnaire developed by Trinidad, Aldridge, and Fraser (2005) for assessing the online learning environment, the criteria of ‘personal relevance’, ‘student interaction and collaboration’ and ‘student autonomy’ have been reached. These criteria are similar to some sub-categories of the ‘programme effectiveness’ factor, a part of which is related to the content, and the other part concerns the individuals’ self-efficacy. Literature emphasises that contents should be based on learning objectives, and the principles of standard content provision should be respected to strengthen the students’ mental abilities. On the other hand, self-efficacy in cyberspace includes the ability to work with computers, search the Internet, and interact and communicate within the LMS (Alqurashi 2016). Indeed, these points, which are considered important in literature, are covered in the ‘programme effectiveness’ factor of EEAM.

The second explored factor in this study is the ‘teaching quality’ which includes teachers’ e-teaching skills such as utilising different virtual teaching methods, providing in-time and suitable feedback, proper timing of assignments and activities and covering the courses within the LMS. In the questionnaire designed by Clayton (2007), ‘ongoing and relevant feedback’ has been referred to in the ‘active learning’ criterion. In Chang and Fisher’s (2001) study, ‘giving feedback and interaction’ have been posed in the criterion of ‘co-participatory activities’. In these two studies, e-teaching skills and the use of virtual training facilities, including forums, online classes etc., were not considered as factors for assessing e-learning environment. One of the reasons for this could be the point that LMS may not have been organised in the current way at that time, and this fact has caused this aspect not to be noticeable in these studies. Meanwhile, nowadays one of the tasks of e-teachers is to prepare content and learning activities, and to provide timely appropriate feedback (Kebritchi, Lipschuetz, and Santiague 2017). Feedback in e-learning is a very important factor in enabling students to monitor and improve their own performance (Yengin et al. 2010). In addition, one of the key elements in e-learning is to interact with others. Students need a variety of interactions with their instructors and peers (Alqurashi 2016; Hung et al. 2010). Therefore, in e-learning, taking advantages of different methods of interaction, such as forums and synchronous online classes, are necessary (Hung et al. 2010). In concurrence with literature, in this study we reached the point that teachers in an e-learning setting had better have the ability and qualifications to recognise and apply proper e-teaching skills to improve quality of teaching.

‘Ethics and professionalism’ is the third driven factor in this study which includes observing the copyright and intellectual property of resources, observing cultural and social issues, the relations governing the educational environment, the responsiveness and availability of teachers, and enhancing the motivation to learn. No study was found in which the observance of intellectual property, cultural and social issues and the governed relationships were considered as factors for assessing educational environment in an e-learning setting. The only found study related to this factor was the study by Clayton (2007), in which the ‘tutor support’ was obtained as an influencing factor. This could be similar to this study’s results in which responsiveness and availability of teachers and being concerned about students’ learning are considered as items for professional and ethical behaviour of teachers. In fact, with their teaching methods, instructors could motivate students and enhance their enthusiasm to study and learn (Kats 2013). In addition, one of the concerns in e-learning is observing the spiritual ownership rights of the owners of the work. Therefore, copyright rules must be respected in loading content and resources. Being honest and respecting the work of others are parts of essential ethical codes. On the other hand, the diversity of culture and the impact of social issues are parts of ethical considerations in an e-learning environment (Toprak et al. 2010). These items that are being emphasised in literature are considered in our study too.

‘Learner support’ is the fourth factor obtained in this study. It includes support for top and weak students, providing academic counselling, accountability of technical and educational staff, access to a suitable digital library, flexible administrative process, and considering students’ concerns on how services are provided to them. In the studies of Jegede, Fraser, and Fisher (1995) and Aldridge, Dorman, and Fraser (2004), the criteria of ‘student support’ and ‘task orientation’ have been proposed. Also, Walker and Fraser (2005) obtained the ‘support of the instructor’ as a factor. It is declared that one of the important and essential services to support e-learners is to have an accountability service to answer questions about educational administrative affairs and technical problems. Also, since most students of virtual programmes are adults and are truly busy in life, flexibility in educational administrative processes such as the maximum permitted length of study and the number of units per semester are among the requirements of an e-learning system (Palmer 2011). It is noteworthy that the learner support system, based on classification, includes three cognitive, affective and systematic domains (Tait 2003). In this study, what have been achieved in this factor are related to affective and systematic domains, meanwhile cognitive ones are covered in factors of ‘programme efficiency’ and ‘teaching quality’.

The fifth factor is ‘safety and convenience’ which includes providing a user-friendly LMS, contents being in harmony with students’ learning styles, not having the feeling of loneliness, and convenience when asking questions. Literature shows the importance of these items. Easy access and user-friendliness of LMS is of great importance to e-students (Stewart et al. 2013). Besides, in an e-learning environment, students feel lonelier and less affiliated. This is one of the challenges that lets down students’ motivation for learning (Baloyi 2014). So, in this study, we found these items as determinants of educational atmosphere in an e-learning setting.

The sixth factor in this study is the ‘awareness of the rules’. It includes awareness of administrative regulations and processes, the existence of clear educational and research guides, and the reputation of virtual education. Jegede, Fraser, and Fisher (1995) and Aldridge, Dorman, and Fraser (2004) obtained the criterion of ‘task orientation’ as one of the determinants for assessing technology-based and distance learning environments respectively. Awareness of administrative regulations and processes was a part of this criterion. In general, it is considered important to provide students with sufficient and clear information about the administrative and educational processes (Baloyi 2014). These are all covered in this study’s instrument.

It is worth mentioning that a study by Chang et al. (2015) could be compared with the present study. They determined the features of technology-supported learning environments based on participants’ perceptions through relevant literature from 1998 to 2014. In their study, technical, content, cognitive, metacognitive, social and affective areas were considered as factors explaining the perception of teachers, students and adult students from an educational environment that has been enhanced with one type of technology. Two domains in Chang et al.’s study, including affective and content areas, are similar to the results obtained in the present study as parts of ‘learner support’ and ‘programme effectiveness’ factors. However, in the study of Chang et al., the goal was not specifically the e-learning environment, and all the technology-enhanced educational setting, for example the use of multimedia alone, had been considered. In contrast, we in this study focused on e-learning environments.

Conclusion

Finally, in this study, EEAM has been developed that could be used to explain the students’ perception of educational environment in an e-learning setting, which is educational atmosphere. This tool is suitable for current e-learning courses which are interactive ones and apply a wide range of synchronous and asynchronous strategies for enhancing teaching–learning processes. EEAM has the advantage of covering relevant aspects and factors explored in previous studies together with new ones, which makes it a comprehensive tool. Using this instrument, educators would explore methods to improve their e-classes based on their students’ perception, leading to enhancing learning outcomes. However, since the theoretical framework of the study had been conducted within the context of virtual postgraduate programmes, that is MSc, it is suggested that additional studies explore more precisely the suitability of applying EEAM for undergraduate context.

Competing interests

The authors declare that no competing interests exist.

References

‘Factors creating an educational atmosphere in cyberspace: a qualitative study’ J. Strides Dev Med Educ, 14: 2017

‘Use of multi trait-multimethod modeling to validate actual and preferred forms of the Technology-Rich Outcomes-ed Learning Environment Inventory (TROFLEI)’ Australian Journal of Educational & Developmental Psychology, 4: 110-125, 2004

‘Self-efficacy in online learning environment: a literature review’ Contemporary Issues in Education Research, 9: 45-52, 2016

‘Critical values for Lawshe’s content validity ratio: revising the original method of calculation’ Measurement and Evaluation in Counselling and Development, 47: 79-86, 2014

‘Learner support in the open distance learning and e-learning context using the community of inquiry model’ Mediterranean Journal of Social Sciences, 5: 1251-1258, 2014

‘Smart learning environment: teacher’s role in assessing classroom attention’ Research in Learning Technology, 27: 2019

‘Virtual learning environments as mediating factors in student satisfaction with teaching and learning in higher education’ Journal of Curriculum and Teaching, 5: 113-123, 2016

‘A review of features of technology-supported learning environments based on participants’ perceptions’ Computers in Human Behavior, 53: 223-237, 2015

‘The validation and application of a new learning environment instrument to evaluate online learning in higher education’ AARE Annual Conference: Paper code. cha01098., Fremantle, 2001

‘The validation of the online learning environment survey’ Proceedings ASCILITE Singapore: 159-167, 2007

‘Developing an instrument to effective factors on clinical learning’ Journal of Advances in Medical Education & Professionalism, 4: 122-129, 2016

‘AMEE Medical Education Guide No. 23 (Part 1): curriculum, environment, climate, quality and change in medical education – a unifying perspective’ Medical Teacher, 23: 337-344, 2001

‘Creating a virtual learning environment’ Journal of Agricultural Informatics, 5: 46-55, 2014

‘Learner readiness for online learning: scale development and student perceptions’ Computers & Education, 55: 1080-1090, 2010

‘The development and validation of a distance and open learning environment scale’ Educational Technology Research and Development, 43: 89-94, 1995

‘Challenges encountered in creating personalized learning activities to suit students learning preferences’ Learning Management Systems and Instructional Design: Best Practices in Online Education: 263-287, Information Science , Hershey, USA, 2013

‘Issues and challenges for teaching successful online courses in higher education: a literature review’ Journal of Educational Technology Systems, 46: 4-29, 2017

‘The relationship of e-learner’s self-regulatory efficacyand perception of e-learning environmental quality’ Computers in Human Behavior, 24: 32-47, 2008

‘The lived experience of flexible education – theory, policy and practice’ Journal of University Teaching & Learning Practice, 8: 1-14, 2011

‘Comparison of health students’ perceptions and expectations of their educational environment’ British Journal of Medicine & Medical Research, 4: 5657-66, 2014

‘Online student support services: a case based on quality frameworks’ Journal of Online Learning and Teaching, 9: 290-303, 2013

‘Reflections on student support in open and distance learning’ International Review of Research in Open and Distance Learning, 4: 1-9, 2003

‘Assessing the efficacy of online teaching with the constructivist on-line learning environment survey’ 9th Annual Teaching Learning Forum: Curtin University of Technology, Perth, 2000

‘Ethics in e-learning’ The Turkish Online Journal of Educational Technology, 9: 78-86, 2010

‘Development, validation and use of the online learning environment survey’ Australasian Journal of Educational Technology, 21: 60-81, 2005

‘Development and validation of an instrument for assessing distance education learning environments in higher education: the distance education learning environments survey (DELES)’ Learning Environments Research, 8: 289-308, 2005

‘Roles of teachers in e-learning: how to engage students & how to get free e-learning and the future’ Procedia Social and Behavioral Sciences, 2: 5775-5787, 2010