Behavior analysis and formative assessments in online oral medicine education during the COVID-19 pandemic

Abstract

BACKGROUND

During the coronavirus disease 2019 (COVID-19) pandemic, traditional teaching methods were disrupted and online teaching became a new topic in education reform and informatization. In this context, it is important to investigate the necessity and effectiveness of online teaching methods for medical students. This study explored stomatology education in China to evaluate the development and challenges facing the field using massive open online courses (MOOCs) for oral medicine education during the pandemic.

AIM

To investigate the current situation and challenges facing stomatology education in China, and to assess the necessity and effectiveness of online teaching methods among medical students.

METHODS

Online courses were developed and offered on personal computers and mobile terminals. Behavioral analysis and formative assessments were conducted to evaluate the learning status of students.

RESULTS

The results showed that most learners had already completed MOOCs and achieved better results. Course behavior analysis and student surveys indicated that students enjoyed the learning experience. However, the development of oral MOOCs during the COVID-19 pandemic faced significant challenges.

CONCLUSION

This study provides insights into the potential of using MOOCs to support online professional learning and future teaching innovation, but emphasizes the need for careful design and positive feedback to ensure their success.

Key Words: Oral medicine, COVID-19, Epidemic prevention and control, Online education, Behavior analysis, Formative assessments

Core Tip: The control coronavirus disease 2019 pandemic disrupted traditional teaching methods, leading to the rise of online teaching. This study explored the effectiveness of massive open online courses (MOOCs) for oral medicine education among medical students in China during the pandemic. Results showed that MOOCs were effective and enjoyed by students, but faced challenges in development. The study highlights the potential of MOOCs for future teaching innovation, but emphasizes the need for careful design and positive feedback.

INTRODUCTION

The urgency to prevent and control coronavirus disease 2019 (COVID-19) caused rapid transition to online teaching, which greatly changed the teaching and learning situations, resources, methods, processes, student evaluation, and other teaching factors[1]. In recent years, online courses, especially massive open online courses (MOOCs), also termed “Mu Ke”, have just become a new educational resource. Since the first introduction of the term MOOC in 2008, the global impact of this teaching model has grown exponentially, with over 101 million global participants, and continues to grow rapidly[2,3]. Presently, the global medical education system is confronted with the task of conducting teaching activities exclusively through online means[4]. In particularly, the integration of information technology with course instruction has highlighted the significance of technology and resources during the COVID-19 pandemic. Online courses possess attributes such as extensive content, abundant information, and versatile formats[5,6]. Some key and difficult knowledge points in teaching can be displayed through tons of pictures and videos. It has become the main form of online teaching. In addition, MOOCs typically include formative assessments that test understanding while maintaining student engagement. Compared with learners of traditional courses, learners who follow these prescribed activities and actively participate in massive open knowledge are more likely to complete the course and achieve better educational outcomes.

In China, the Mu Ke platform was launched in 2013 by XYZ university. Since then, several Chinese universities have launched multiple platforms and courses that are typical of MOOCs[7,8]. The rapid growth of MOOCs in China has been accelerated by the recent COVID-19 pandemic. In contrast to conventional general curriculum education, medical MOOCs have gained prominence within higher education. The application of MOOCs in diverse educational settings has also opened doors for the integration of information technology in classrooms, curricula, and teaching approaches[9]. Although MOOC presents a wide range of themes, it continues to be questioned[10-15]. Several factors that differ from traditional teaching methods may limit MOOCs. For instance, most medical MOOCs have been migrated from previous traditional courses, with nearly half only consisting of course software and text information. Secondly, although stomatology is generally considered a highly specialized field, it has not received considerable attention in the curriculum[16-19]. Here, we conducted a comprehensive survey to investigate the MOOC status in stomatology in China through the analysis of the course offering data, learning behavior data, and student surveys[20].

This study aimed to investigate the effectiveness of online teaching in oral medicine, which focuses on the introduction to oral science, available on the website platform and mobile devices since 2017[21-27]. Our findings may reflect the benefit of using the MOOC format to disseminate knowledge in oral medicine and study the demographics and course engagement of learners thus far[28,29]. Specifically, the efficacy of and feedback on the online course was assessed. Unlike existing online learning evaluation methods ¾ which only consider test and assignment scores ¾ our survey comprehensively considers multiple learning behaviors of the learners, such as viewing videos, doing exercises, taking exams, and participating in discussions. This approach provides a bigger picture of students’ online learning experiences.

MATERIALS AND METHODS

Design of oral medicine online courses

MOOCs are divided into 11 chapters, including oral anatomy and physiology, dental endodontics, maxillofacial surgery, prosthodontics, and orthodontics. A total of 33 videos are available online, with a duration of 370 min. Ten of the 33 videos are less than 10 min long, accounting for 30.3%, and 23 are between 10 and 20 min long, accounting for 69.7%. Additional videos that do not contain learning content, such as course introductions, were not included in the statistics.

This online course was designed by Wenzhou Medical University, the Third Military Medical University in partnership with credit course operation service platform (www.zhihuishu.com) to provide a basic understanding of dentistry, review areas of dental medicine, introduce components of oral disease and treatment, and review some areas of dental research. In keeping up with the outlined principles of five-year undergraduate medical education, online courses allow medical students to complete the modules and gain corresponding credits. The target audience includes students majoring in basic medicine, clinical medicine, preventive medicine, and radiation medicine. These students can use the website or mobile phones to download the “teacher circle” app to study online.

Operation mode of online courses

Since its launch, courses have been continuously available on the website for personal computers and the “teacher circle” app (for mobiles) for learner enrollment at corresponding time points. The content provided in the online courses was arranged into 7 wk, each following a theme. Each week included multiple short videos (8-20 mins in length) on the week’s theme. Each video also included embedded multiple choices and true/false quizzes to test learners’ understanding of the material presented and maintain engagement with the presentation. At the end of each chapter, learners were asked to complete the unit test on the knowledge presented in that week’s lectures. Given the universal availability of MOOC and referring to the traditional teaching model, students were awarded credit upon completion of the course. To keep online teaching interactive, the design also included discussion boards for learners throughout the course. These interactive links included forum presentations, assignments, and general discussions, where students can ask or answer questions for each other. The board was monitored by a course staff, who responded to all posts and assisted learners with any inquiries.

Basic information about online teaching

To assess the impact of online teaching, we analyzed the data related to participating schools, learner engagement, and enrollment. Our study cohort was learners who chose oral medicine online course as an elective course at the school, and those who had already completed relevant online learning courses and passed the examinations. We provide a summary of basic analytics, such as school numbers, enrolment numbers, and active learners throughout the course. Online course behavior analysis was performed and student survey data and feedback were evaluated to further assess the students’ online learning situation.

Online course behavior analysis

The statistical data of online course behavior analysis included the completion rate of MOOC, average score, number of qualified students, and number of excellent students. Grade and score statistics were divided into unqualified (score < 60), qualified (score > 60), and excellent (score > 90). The differences between data were analyzed by comparing relevant data on the platform. To analyze students’ online learning behavior and demonstrate consistent participation during the study, professional staff collected online learning time data, including the average viewing time, video length, and learning time ratio. The learning time ratio equals the average viewing time of the video divided by the video length. Moreover, the contents of the 11 chapters were divided into five categories: “Oral anatomy and physiology”, “dental endodontics”, “periodontology”, “maxillofacial surgery” and “prosthodontics and orthodontics”. The learning time ratio in different chapters can indirectly reflect the quality of chapter design, which promotes further improvement future studies. The differences in content structures, professional staff, online/offline activities, and international certificates were compared.

Student survey data and feedback

The survey was based on students who had completed their online learning courses. Due to the COVID-19 pandemic, we created a responsive educational ecosystem by enlisting educators and learners to be supported by various learning modalities, including the delivery of educational content through e-learning methods. Therefore, the application rate of mobile devices in online courses was also analyzed. To assess the impact of the course from the learners’ perspective, overall satisfaction with the curriculum ranged from “strongly agreed”, “agreed”, and “neutral” to “disagreed”. Further information on the progression of learners throughout the course was made available to the course team at the end of the course. To objectively analyze the student survey data and feedback, we used structured questionnaire analysis, which includes a comprehensive evaluation, instructional design, teacher guidance, online communication, and network situation. The structured questionnaire consisted of positive statements designed to determine satisfaction with the learning experience.

RESULTS

Basic information about online teaching

A total of 15 schools offered elective courses for 971 students, of which 922 students enrolled. A total of 768 students had completed the teaching plan according to the course standards in 13 schools. Therefore, data analysis was based on the 768 students in 13 schools (Figure 1A). An important limitation of this research is that no data was available on students who registered but did not complete the course according to the unified standard (two schools had missing data). As an elective course, about 88% of learners had completed the curriculum. This rate was significantly higher than that of general network courses (XYZ%) but significantly lower than that of the professional elective course (94.4%) on the platform. Analysis of academic records showed that about 33% of the students achieved an excellent score and 85.9% passed the exam (Figure 1B). To further understand the difficulty in the curriculum and knowledge acquisition by students, the average score, the passing rate, and the excellent rate of oral medicine and other medical courses on the platform were compared horizontally (Figure 2). The results revealed that the rating of online teaching of oral medicine was lower than that of medical courses. Given the specialty feature of oral medicine, we believe that this result is reasonable.

Figure 1 Different states in the enrolled 13 schools. A: Online learning status in the enrolled 13 schools; B: Academic record in the enrolled 13 schools.

Figure 2 Comparative analysis of stomatology and other massive open online courses in the platform.

Online course behavior analysis

The degree of students’ participation is an important indicator of students’ behavior analysis. It not only reflects students’ learning interests, learning enthusiasm, and motivation but also indirectly reflects the quality of curriculum design. Too short a viewing time is insufficient for learning. As shown in Figure 3A, the average viewing time of students in different schools was satisfactory. The total length of the course was 370 min, which also reached 282 min even in schools with the lowest completion rate. The ratio of average viewing time fluctuated between 76.2% and 101.9%. In addition, some key statistical results (e.g., the time length of the online sessions in different sections) that were collected from students for the instructors to learn the pros and cons of the online chapters are presented in Figure 3B. Behavior analysis exhibited a regular wavy graph, which peaks in the introduction of the chapter and hits rock bottom at XYZ, especially 7-4 (fracture of oral and maxillofacial regions) and 9-3 (benign and malignant tumors) involving numerous professional jargons and clinical skills, which are difficult for medical students to understand.

Figure 3 Behavior analysis. A: Behavior analysis of stomatology massive open online courses (the video learning ratio); B: Behavior analysis of stomatology massive open online courses (video learning in champers).

Student survey data and feedback

The course evaluation was completed by 276 learners (Figure 4). Over 90% of respondents agreed or strongly agreed that the learning experience was helpful, of which 67.4% “strongly agreed”, 2.9% held a “neutral” opinion, and 2.9% “disagreed”. The feedback from learners showed that the overall satisfaction with the curriculum was 94.6% (“strongly agreed” and “greed” included). Both ratings and written feedback reflected students’ satisfaction with the experience. Most learners believed that the course was all-rounded, enlightening, and rewarding. The comprehensive evaluation revealed that 92.8% of learners did not regret choosing this course, learners were satisfied with the study and were willing to recommend the courses to others. In the instructional design survey, 94.6% of learners agreed that the videos of the online course were of high quality, the design of the course content was reasonable, and the course was well-taught, which actively encouraged students to participate. In the teacher guidance survey, 90.9% of learners agreed that learning the course was helpful. For online communication and network situations, 92% of learners were unanimous that the online forum communication improved their course study, and the online question answering was timely and effective. Besides, 94.2% of learners agreed that viewing online course videos was very smooth and sound, and pictures were very clear.

Figure 4 Formative assessments in stomatology massive open online courses.

Some students proposed important suggestions, and the most frequent comments are presented in label 1. These comments reflected the full range of students’ experiences in an online course. Based on the most common responses to the course evaluation as shown in Table 1, participants were asked to suggest the aspects of MOOCs that did work well, and the result showed that most students believed that online courses provide positive experiences.

Table 1 Representative feedback.

	No.	Representative feedback on stomatology online teaching
Student evaluation	1	“It helps me to realize the importance of the oral cavity. It also gives me a deep understanding of oral hygiene and protection, as well as my wisdom teeth”
	2	“Getting to know the specific structure of teeth and some complications of oral diseases has made me fully understand various oral diseases and preventive measures. At the same time, through the study of the course, I realized how important knowledge is”
	3	“I learned some knowledge about the oral cavity, especially about the oral anatomy and common diseases. I cherish my teeth better after knowing more about them”
	4	“Getting to know oral science, and gaining some knowledge of it helps me better maintain my oral hygiene and protect my teeth”
Recommendations and expectations	5	“I enriched myself with the knowledge of the tooth structure and some diseases and learned how to treat many oral diseases and clean up the oral cavity, which also push me to pay more attention to personal oral hygiene”
	6	“The teaching of some oral problems is too general to be understood. I hope it can be more specific and detailed. I hope the exercises will not be too technical, which are too difficult for students who are not majoring in the subject”

DISCUSSION

The conclusions reached in the present study were based on data obtained by the Eastern Western University curriculum sharing alliance and the www.zhihuishu.com platform. The study results demonstrated that teaching methods are very suitable for the theory of the Mu Ke form. Different schools and learners may have differences. However, the main difference comes from the teaching model of online courses, which is particularly important in awarding credits. The most notable difference between professional and regular courses is that for regular courses, the participation rate of about 25% is already impressive. The low retention rate of MOOCs has been widely documented. Some people believe that low retention rates of MOOC limit its value compared with traditional face-to-face teaching[30]. Although the ratings of our professional courses (completion rate of 88%) are still lower than those of professional elective courses on the platform (94.4%), students are willing to accept this form of learning. Our research results indicate that if academic institutions include massive open oline courses knowledge in their curriculum and its results as part of formal student evaluations, participation and completion rates may be further improved. Notably, the results of the completion rate analysis do not show the effectiveness of online courses for learning, but they only highlight the willingness of students to participate in this learning style. Interestingly, learning behavior data showed that most learners watched videos, completed exercises, and participated in discussions through their mobile phones. This implies that the online education mode is not limited to the traditional indoor PC mode. The mobile mode is more popular, and learners can study online anytime and anywhere, including at home, in the park, on the subway, etc. Learning enthusiasm is greatly affected by communication and convenience. For example, N06’s school is located in Tibet ¾ a remote area in western China. Poor communication limits the effectiveness of online teaching and learner motivation, with a low school completion rate (69.1%) and a low pass rate (64.2%). The provision of video learning ratios also supports this conclusion. The diversity of learning styles makes the students’ learning time significantly longer than in other classes.

The shift from traditional courses to Mu Ke education has also led to changes in students’ learning motivation. The initial goal of most learners is only to obtain a course certificate. However, as learning deepens, most learners are attracted by the rich learning content and online videos. Due to the special teaching methods of MOOC, along with changes in learners’ motivation, it is necessary to conduct a comprehensive assessment of learning[31]. We compared online courses in stomatology with similar courses on the same platform. According to the survey, the difference between online teaching of stomatology and online teaching of other medical courses cannot be ignored. Stomatology is different from other medical disciplines, which have relatively high requirements for online courses. Behavioral analysis and student survey data provide further insights into the learning experience of online courses. The average viewing time means that the openness of MOOCs allows learners to return to the course multiple times[32]. They return to the course when needed and refresh specific topics when they need to solve problems in their learning practices. In addition, studying the learning engagement of students will reveal new insights into their motivation and learning process, and provide clues to the design of future MOOCs and alternative online teaching approaches[33].

Goals should be the driving force behind a series of assessments in which students will demonstrate learning progress. Generally, these goals are used to guide the formative and summative framework to form a complete evaluation system for schools[34]. However, in this special period, formative assessment in dentistry should be emphasized because it provides more responsive feedback, with a focus on quality improvement and professional development rather than scores or ratings. These data stimulate a sense of cautious optimism and raise the question of whether online courses can indeed play a crucial role in the prevention and control of the COVID-19 pandemic. In addition to learning theoretical knowledge, training in clinical practice and clinical skills is also very important for learners[35]. One participant stated that “some medical courses should undergo technical training in a real environment. It is more difficult to reliably reproduce the content provided by virtual online materials”. Nevertheless, there are still some challenges to overcome. Another learner commented that “The course will be more vivid, humorous, and interesting, maintaining the interaction between teachers and students, and thereby encouraging students to grasp this knowledge more solidly through more practical cases”. Some medical courses involving practice ¾ which virtual online teaching cannot provide ¾ should be conducted in a real environment[31,36].

However, this study still has certain limitations. Firstly, we only analyzed regional research data, and information were obtained from a single hospital, which may lead to biased results. In the future, data from different countries, regions, and age groups should be analyzed to eliminate contingency and address the aforementioned limitations. In addition, follow-up studies should enrol a larger sample size and increase the scope of oral education to cover subjects such as educational forms and educational models.

CONCLUSION

In summary, Mu Ke will never completely replace skilled teachers who provide face-to-face education in the classroom. Mu Ke also does not provide practical experience for students. However, this may be the best teaching method in unprecedented situations, such as during the COVID-19 pandemic, which may provide a consistent learning experience for a large number of students from different regions of China[37-39]. We believe that MOOCs can potentially provide medical education for students with a free, convenient, enjoyable, and engaging educational experience.

ARTICLE HIGHLIGHTS

Research background

Coronavirus disease 2019 (COVID-19) disrupted traditional teaching, making online education crucial. The effectiveness of online teaching among medical students needs examination.

Research motivation

This study aims to investigate the challenges and effectiveness of using massive open online courses (MOOCs) for oral medicine education in China during COVID-19.

Research objectives

This study aims to assess the current status and challenges of stomatology education in China, and to evaluate the necessity and effectiveness of online teaching methods among medical students using behavior analysis and formative assessments.

Research methods

Online courses were developed and delivered on personal computers and mobile devices. Behavior analysis and formative assessments were used to evaluate student learning status. Data from course behavior analysis and student surveys were also collected.

Research results

Most learners completed MOOCs and achieved better results. Students enjoyed the learning experience, but the development of oral MOOCs during COVID-19 faced significant challenges.

Research conclusions

MOOCs have potential for supporting online professional learning, but require careful design and feedback to ensure success. Online teaching needs to be better integrated with medical education in the future.

Research perspectives

Future research could explore how to better integrate online teaching with medical education and enhance the design and effectiveness of MOOCs for oral medicine education.