This paper is in the following e-collection/theme issue:

Nursing Education and Training (113) e-Learning and Medical Education (658) mHealth in Medical Education and Training (67) Undergraduate Education for Allied Health Professionals (37)

Published on in Vol 3, No 1 (2020): Jan-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/19780, first published .
Utilization of Nursing Education Progressive Web Application (NEPWA) Media in an Education and Health Promotion Course Using Gagne’s Model of Instructional Design on Nursing Students: Quantitative Research and Development Study

Utilization of Nursing Education Progressive Web Application (NEPWA) Media in an Education and Health Promotion Course Using Gagne’s Model of Instructional Design on Nursing Students: Quantitative Research and Development Study

Utilization of Nursing Education Progressive Web Application (NEPWA) Media in an Education and Health Promotion Course Using Gagne’s Model of Instructional Design on Nursing Students: Quantitative Research and Development Study

Authors of this article:

Deny Yuliawan1 Author Orcid Image ;   Doni Widyandana1 Author Orcid Image ;   Rachmadya Nur Hidayah1 Author Orcid Image
Article Authors Cited by (5) Tweetations (1) Metrics

    Original Paper

    Department of Medical and Health Professions Education, Faculty of Medicine, Public Health and Nursing, University Gadjah Mada, Yogyakarta, Indonesia

    Corresponding Author:

    Deny Yuliawan, RN, MHPE

    Department of Medical and Health Professions Education

    Faculty of Medicine, Public Health and Nursing

    University Gadjah Mada

    Radioputro Building 6th Floor, Jl. Farmako, Sekip Utara

    Yogyakarta, 55281

    Indonesia

    Phone: 62 0274562139

    Email: denyyuliawan92@gmail.com


    Background: Previous studies have proven that web-based learning media that offer interesting features with the learning management system concept could support the learning processes of nursing students. Nonetheless, it is still necessary to conduct further research on its potential as an information media that supports learning using 1 of the mobile learning methods.

    Objective: This study aims to develop and use the Nursing Education Progressive Web Application (NEPWA) media in an education and health promotion course for nursing students.

    Methods: This is a research and development study aimed at developing the NEPWA media using the Analyze, Design, Develop, Implement, and Evaluate approach and a quantitative research with descriptive and pre-experimental 1-group pretest-posttest design conducted in the Study Program of Nursing Sciences, Faculty of Health Sciences, Muhammadiyah University of Surakarta. A total of 39 nursing students in their second year of undergraduate studies participated in this study. A pretest-posttest design was used to measure any changes in the dependent variable, whereas a posttest design was used to measure any changes in the independent variables.

    Results: After using the NEPWA media, there was a significant increase in the student knowledge variable (N=39; knowledge: P<.001; 95% CI 23.88-33.14). In terms of student satisfaction with the learning process using Gagne’s model of instructional design, most of the students were satisfied, with a mean score of ≥3. In addition, the results of the measurement using the System Usability Scale on the NEPWA media showed that NEPWA has good usability and it is acceptable by users, with a mean score of 72.24 (SD 8.54).

    Conclusions: The NEPWA media can be accepted by users and has good usability, and this media is designed to enhance student knowledge.

    JMIR Nursing 2020;3(1):e19780

    doi:10.2196/19780

    Keywords

    mobile application; nursing students; blended learning; knowledge; satisfaction; system usability; mobile phone 


    Learning in the age of digital transformation has different characteristics compared with that in the past decades. The learning model in the digital age has created a new trend that uses electronic media known as mobile learning (m-learning), which offers many benefits such as flexible and portable learning designed without space-time constraints [1].

    In general, the majority of internet users are middle and high school or university students aged 10-24 (18.40%) years and 25-34 (24.40%) years [2], respectively, whereas in the context of health sciences, the ages of 10-18 years are a period of preparation before individuals enter higher education, where, in this age group, the majority of internet users are at the high school education level. One of the unique features of web-based learning is the availability of an independent, effective, and efficient learning environment where users can repeatedly access information and skills to practice without space-time constraints [3], which could benefit the learning in both campuses and clinics [4].

    Students are interested in using web-based learning through mobile apps as they offer many interesting features [5-8]: mobile apps can be installed in every Android smartphone [9] and are flexible to use. These are the factors that attract users [10]. Learning media using m-learning, which can be applied in lectures using demonstration learning methods to become more interactive, are popular and commonly used in the health sector [1]. On the other hand, 1 of the limitations of traditional lecture methods is that they can be boring as lecturers deliver materials only via texts, making the lectures passive. Thus, it is necessary to address this issue by using effective learning media, one of which is the use of web-based learning media in the form of mobile apps with a simple concept of a learning management system that could stimulate interactions during lectures and provide feedback to students; therefore, it is necessary to design learning strategies well [1].

    This is in line with some previous studies, which showed that app media that integrate web-based lectures using electronic learning (e-learning) media can increase student knowledge about the given courses [11-13]. On the basis of a preliminary study by interviewing one of the lecturers and students at the Study Program of Nursing Science, the Faculty of Health Sciences, Muhammadiyah University of Surakarta, it can be concluded that some of the learning media used by lecturers are in the form of PowerPoint slides, handouts, and videos from YouTube, as well as the use of one of the principles of Gagne’s instructional design, that is, pretest. Thus, it is necessary to innovate web-based learning media to be used during lectures to assist students in studying.

    The interactive learning strategies in this lecture were designed using the 9-step Gagne’s model of instructional design with a teaching method using blended learning. Gagne's model of instructional design includes instructions that allow learning processes to be effective [14,15].

    On the basis of the abovementioned points, the problems of this study were formulated as follows: “Do the development and utilization of the NEPWA media on education and health promotion course influence the improvement of nursing students’ knowledge?”

    The objective of the study was to develop and utilize the Nursing Education Progressive Web Application (NEPWA) media in an education and health promotion course among nursing students.


    This was a research and development study [16]. The development of the NEPWA media used the Analyze, Design, Develop, Implement, and Evaluate approach [17]. The first needs analysis was conducted according to the needs of the researchers to develop apps and websites in the form of both software and hardware as well as materials in the NEPWA media. Afterward, the NEPWA media were designed in accordance with the needs of lectures by adjusting to Gagne’s model of instructional design, including the features of learning material rooms, discussion rooms, assignment rooms, and test rooms. The design for NEPWA media was then developed and made by a team expert app and web programmer based on the researchers’ designs. The web-based learning media created was then named NEPWA. The NEPWA media was developed into 3 sections with their own features, which were correlated with one another, available in the form of both websites and apps. The first result of the NEPWA media was student page, which consists classrooms, discussion rooms, assignment rooms, and test rooms and user profile features, as shown in Figure 1.

    Next, students can use the classroom feature that contains instructions on how to use the NEPWA media, modules, course units, and learning materials in the forms of PowerPoint slides, infographics, and the resume of learning materials, as shown in Figure 2.

    In addition, students’ test room features are also available, as shown in Figure 3.

    ‎
    Figure 1. Display of the student profile page on the Nursing Education Progressive Web Application media.
    View this figure
    ‎
    Figure 2. Display of the student classroom page on the Nursing Education Progressive Web Application media.
    View this figure
    ‎
    Figure 3. Display of the test room page on the Nursing Education Progressive Web Application media.
    View this figure

    The strengths of the test room in the NEPWA media are, first, the questions that simultaneously show up on the account of 1 student are not the same as those showing up on the accounts of other students; thus, students could focus on the questions showing up on their own page. Second, the duration and number of questions set by lecturers are also displayed. Third, the scores for each of the tests taken by students will show up at the end of the test session as taken by each student.

    For the second results, the development of the lecturer page display on the NEPWA media contains the features of the question room, test room, test score, learning material room, assignment room, and discussion room, as shown in Figure 4.

    ‎
    Figure 4. Display of the lecturer page containing the question bank and list of tests on the Nursing Education Progressive Web Application media.
    View this figure

    The benefits are question rooms for each course and examination registration room for students that can be managed directly by the lecturers are available, such as setting the name of the examination, the number of questions tested, and the examination duration. In addition, there are also learning outcome rooms that display the scores of all students who take each examination.

    For the third result, the development of the admin page on the NEPWA media consisted of the features of student data, lecturer data, course description data, questions, and the test results, as shown in Figure 5.

    The admin page displays student data that can only be managed directly by 1 administrator. The data include student name and number, study program, and automatic username-password to be used by students to log in to their accounts on the NEPWA media. In addition, there are also features of lecturer data and course description data, as seen in Figure 6.

    ‎
    Figure 5. Display of the administrator page containing student data on the Nursing Education Progressive Web Application media.
    View this figure
    ‎
    Figure 6. Display of the admin page containing data on lecturers and courses on the Nursing Education Progressive Web Application media.
    View this figure

    The features of lecturer data and course description data that can only be managed directly by 1 administrator are available. The data include the lecturer name and automatic username-password to be used by lecturers to log in to their account on the NEPWA media; these are linked to the features of course description. Moreover, the feature of course description data can be inputted by the admin by entering the name of the course that has been approved by the study program.

    The next step is the implementation of the NEPWA media that had been created. The NEPWA media was tested and rated by the students. Once the respondents tried using the NEPWA media, they were then asked to provide feedback by filling out a questionnaire to measure the System Usability Scale (SUS) of the NEPWA media, which was conducted after the posttest was completed.

    The next step was when the NEPWA media was already tested and ready to use. Subsequently, research using the NEPWA media was conducted. This was a quantitative study with descriptive and pre-experimental one-group pretest-posttest designs [16,18,19], presented in Figure 7. The group descriptions in Figure 7 are as follows:

    • O1: Pretest in the sample group
    • X: Treatment (NEPWA media)
    • O2: Posttest in the sample group
    ‎
    Figure 7. Scheme 1 Quantitative study with descriptive and pre-experimental 1-group pretest-posttest design.
    View this figure

    The following 3 aspects were measured:

    1. Student learning outcomes in the factual knowledge (cognitive) domain were measured using multiple-choice questions before and after the face-to-face session as the pretest and posttest in the sample group.
    2. Satisfaction with Gagne’s model of instructional design was measured after students completed the face-to-face and posttest sessions. The satisfaction questionnaire was adopted from the study by Buzzetto More [20] and used in a previous study by Fandianta [13] in face-to-face and web-based sessions (Multimedia Appendix 1).
    3. SUS was measured after students completed the face-to-face and posttest sessions using a reliable and validated questionnaire by Sharfina and Santoso [21] and Mohamad Marzuki et al [22] (Multimedia Appendix 2).

    In this study, researchers used a sample of undergraduate nursing students (N=39) calculated using the Harry King’s nomogram with saturation [16,23] in the fourth semester of the second year at the Muhammadiyah University of Surakarta between June 2019 and July 2019. These students were asked to participate in this study, and no one refused. No student was excluded, as all of them met the important inclusion criteria of the study, that is, being registered as active students in the education and health promotion course and having a smart phone or tablet to access the NEPWA media. The study had 1 variable, namely, the independent variable that measured satisfaction with Gagne’s model of instructional design through the NEPWA media using a satisfaction questionnaire [20] in the Indonesian version [13] adjusted to Gagne’s model of instructional design [14,15], and the usability of the NEPWA media was measured using the SUS score [24] in the Indonesian version [21] and the Malay language version [22]. The dependent variable then measured student knowledge using 50 multiple-choice questions with cognitive skill level C2 according to Bloom’s taxonomy. Meanwhile, the data collection technique used printed paper for the SUS questionnaire; the NEPWA media was used for the results of the pretest-posttest. This study consisted of 3 stages.

    The preparation stage lasted 4.5 months, where the researchers had coordinated with lecturers during the process of designing the module as well as pretest-posttest questions to test student knowledge. In addition to these activities, the researchers also coordinated with the programming team during the process of designing the NEPWA media. Once completed, the researchers pilot tested the NEPWA media, which contained a module to the supervisors, some students of Master of Medical and Health Profession Education and students of Magister of Nursing programs who were in their final semester. Finally, the implementation stage lasted 2 weeks, as shown in Figure 8.

    ‎
    Figure 8. Scheme 2 Research Implementation.
    View this figure

    Once all the data were collected and entered into Microsoft Excel, analysis of the characteristics of the research subjects and interpretation of the NEPWA media system usability using SUS score [25,26] were conducted. Mean SUS score ≥68 is categorized as having good usability [21,22,25]. The next step was to conduct an analysis of satisfaction with Gagne’s model of instructional design. The difference between the pretest-posttest scores in terms of the students’ factual knowledge was then determined by analyzing the data of score difference that had normal distribution. Thus, differences could be tested using a parametric statistical test, that is, the paired sample t test.

    In conducting this study, the researchers obtained research ethics approval from the Committee of Research, Medical and Health Ethics, Faculty of Medicine, Public Health and Nursing at Gadjah Mada University in Yogyakarta with Ref: KE/FK/0290/EC/2018, dated March 18, 2019.


    NEPWA Media

    The learning media developed by the researchers and a team of programmers ran well. The web-based learning media, called NEPWA, is accessible through the website [27]. It can also be installed as an Android app.

    Characteristics of the Subjects

    The research could be implemented according to the research design. The students participated in this study voluntarily after they were given an explanation about the research objectives and processes. The selection of the students as the research participants took place right before the pretest was completed after a lecture session on education and health promotion ended, starting on June 17, 2019, in 1 class. There were 39 students in 1 classroom, and 100% (39/39) of them agreed to be the research subjects, and they all met the inclusion criteria. Consequently, these 39 students fully participated in the research activities, and the results were further analyzed.

    The characteristics of the research subjects were in the form of demographic data, including age, sex, and grade point average. The distribution normality of the participants’ characteristic data was analyzed using the Shapiro-Wilk criteria: data are said to be normally distributed if the P>.05. The detailed characteristics of the respondents are shown in Table 1.

    Table 1. Characteristics of the respondents (N=39).
    CharacteristicsaRespondents
    Age (years), median (min-max)20 (19-21)
    Grade point average, mean (SD)3.54 (0.17)
    Sex, n (%)
    Male6 (15)
    Female33 (85)
    Frequency of using web-based learning media during lectures, n (%)
    Not using at all1 (3)
    <30 min8 (21)
    30-60 min21 (54)
    1-2 hours8 (21)
    3-4 hours1 (3)

    aData collection process was carried out in 2019 using the first or primary source.

    Student Satisfaction With the Use of NEPWA Media

    The survey of satisfaction with the NEPWA media filled out by student participants included media used to access the internet (mobile phones, laptops, tablets, and personal computers), internet access (Wi-Fi and cellular data), and the frequency of using instructional media. The descriptive analysis results indicated a good level of satisfaction with the use of the NEPWA media. In fact, most of the students used mobile phones as hardware and cellular data to access information provided on the NEPWA media. Besides, the majority of them (>40%) accessed the NEPWA media 1-2 times per day for <30 min before, during, and after the education and health promotion course took place. In addition, the most commonly used and the most useful features were the learning material room and the test room, with a frequency >64%. Meanwhile, the brief answers on the satisfaction questionnaire regarding the use of the NEPWA media resulted in some written statements from the participants, with 3 questions that covered several subthemes of the resume, as follows:

    Good aspects of NEPWA media

    • Learning has become more effective, efficient, time-saving, and simpler because it can be done anytime and anywhere.
    • The learning materials can be read repeatedly, making it easier for students to study.
    • It is easier to use all available features.

    Some aspects of NEPWA media that need to be improved

    • It is necessary to perform system monitoring, preventing it from being disrupted easily.
    • The display should be made more attractive, especially the display of user profile, learning material room, and discussion room.

    Satisfaction with the use of NEPWA media

    • Availability of many test practice questions.
    • Availability of comprehensive learning materials.

    SUS of NEPWA Media

    On the basis of the results of the distribution analysis, the NEPWA media had good usability and could be accepted by the users in this study, that is, the measurement uses the system usability variable of the NEPWA media of all students (N=39). The mean SUS score per statement can be seen in Table 2.

    Table 2. Mean System Usability Scale score for each statement.
    NumberStatement itemsaValues, mean (SD)
    1I think that I would like to use this system3.03 (0.28)
    2I found the system unnecessarily complex2.95 (0.51)
    3I thought the system was easy to use3.05 (0.39)
    4I think that I would need the support of a technical person to be able to use this system2.56 (0.91)
    5I found that the various functions of the system were well integrated3.15 (0.43)
    6I thought there was too much inconsistency in this system2.87 (0.52)
    7I would imagine that most people would learn to use this system quickly3.05 (0.61)
    8I found the system very cumbersome to use3.03 (0.49)
    9I felt very confident using the system3.03 (0.67)
    10I needed to learn a lot of things before I could get going with this system2.18 (1.02)

    aData collection process was carried out in 2019 using the first or primary source.

    Table 2 presents the mean score for each of the statements of all the subjects, and it can be seen that the positive statements 5, 3, 7, 1, and 9 had a mean score of 3 (ranging from 0 “strongly disagree” to 4 “strongly agree”), which tended to be higher than the negative statements. Next, the total score of each respondent was multiplied by 2.5, resulting in the SUS score of each respondent. The total SUS score was obtained by summing the SUS scores of all the respondents, resulting in a total score of 2817. Afterward, the mean SUS score was obtained by dividing the total SUS score by the number of respondents, resulting in a score of 72.24 (SD 8.54), which indicated that the NEPWA media has good usability, meaning that it is below the acceptable range for the user, as shown in Figure 9.

    ‎
    Figure 9. Results of System Usability Scale score on the Nursing Education Progressive Web Application media. SUS: System Usability Scale.
    View this figure

    Satisfaction Scale With Gagne’s Model of Instructional Design

    On the basis of the results of the satisfaction analysis, the majority of the respondents in this study were satisfied with the use of Gagne’s model of instructional design in the course using the NEPWA media from all the students (N=39). The mean score of satisfaction with Gagne's model of instructional design using the NEPWA media per statement item is presented in Table 3.

    Table 3. Mean score of satisfaction with Gagne’s model of instructional design using the Nursing Education Progressive Web Application media for each statement.
    NumberStatement itemaValues, mean (SD)
    1Overall, I am satisfied with learning that uses this teaching media2.92 (0.58)
    2I am satisfied with learning that uses this teaching media as a source of information2.97 (0.54)
    3I am satisfied with the contents available in this learning media2.95 (0.51)
    4I always use this learning media to answer questions2.62 (0.71)
    5I am satisfied with the contents or materials available in this learning media to assist me in learning3.15 (0.63)
    6I believe that this learning media could enhance my understanding3.00 (0.51)
    7I am satisfied with the brief summary available in the PowerPoint slides in this learning media3.15 (0.49)
    8I feel that this learning media provides instructions/keywords available2.79 (0.73)
    9I am satisfied that I can have discussions using this learning media3.00 (0.51)
    10I feel that the web-based discussion room available in this learning media could help me gain understanding of learning materials2.97 (0.43)
    11I am satisfied that I can interact with lecturers and other students through the web-based discussion rooms available in this learning media2.92 (0.62)
    12I am satisfied that I could take tests using this learning media3.08 (0.42)
    13I feel that the web-based tests available in this learning media are easier to use3.03 (0.54)
    14I am satisfied when I receive scores immediately after taking tests using this learning media3.03 (0.43)
    15I am satisfied that I can send assignments on the web using this learning media3.00 (0.46)

    aData collection process was carried out in 2019 using the first or primary source.

    On the basis of the results from Table 3, it can be seen that the mean score of the statement items was ≥3, indicating that the students were satisfied with several statement items or features available in the NEPWA media using Gagne’s model of instructional design, including:

    1. Satisfaction with the modules or materials available in the learning media.
    2. Satisfaction with the brief summary on the PowerPoint slides available in the learning media.
    3. Satisfaction because they could take tests using the learning media.
    4. Satisfaction because they could easily take web-based tests using the learning media.
    5. Satisfaction because they received a score immediately after they completed a test using the learning media.
    6. Satisfaction because they could send assignments on the web through the learning media.
    7. Satisfaction because they could have discussions using the learning media.

    Knowledge-Level Distribution Analysis on the NEPWA Media

    The results from the analysis showed that the data were normally distributed. There was an increase in the variable of knowledge level using the NEPWA media among all the students (N=39). The increase in the mean score before and after using the NEPWA media is presented in Table 4.

    Table 4. Results of analysis of knowledge levels on the Nursing Education Progressive Web Application media.
    ResultsaPretest, mean (SD)95% CIPosttest, mean (SD)95% CI
    Knowledge levels44.41 (12.36)40.40-48.4272.92 (14.62)68.18-77.66

    aData collection process was carried out in 2019 using the first or primary source.

    On the basis of the results in Table 4, it can be seen that the pretest mean score was 44.41 (SD 12.36), which increased to 72.92 (SD 14.62) in the posttest after using the NEPWA media. This indicates that there was an increase in the mean score where the posttest score was higher than the pretest score.

    Analysis of Differences in Student Knowledge Improvement Using the NEPWA Media

    The level of student knowledge before and after using the NEPWA media and the difference between these two were found to have a normal data distribution; thus, the analysis used a paired sample t test to determine the significance of the changes in the level of student knowledge before and after using the NEPWA media. From the test results, the mean score of student knowledge before using the NEPWA media was 44.41 (SD 12.36), which significantly increased (P<.001) by 23.51 (SD 14.29) to 72.92 (SD 14.62) after using the NEPWA media. The detailed results of the paired sample t test on the level of student knowledge are presented in Table 5.

    Table 5. Differences in the levels of student knowledge before and after using the Nursing Education Progressive Web Application media analyzed through paired t test.
    Levels of student knowledgeaNMean (SD)Difference in mean (SD)95% CIP value
    Before using the NEPWAb media3944.41 (12.36)28.51 (14.29)23.88-33.14<.001
    After using the NEPWA media3972.92 (14.62)N/AcN/AN/A

    aData collection process was carried out in 2019 using the first or primary source.

    bNEPWA: Nursing Education Progressive Web Application.

    cN/A: not applicable.

    Table 5 shows that the P value(<.001) is less than the α value (.05); thus, it can be concluded that there were significant changes in the score of knowledge of education and health promotion before and after using the NEPWA media. These results were also supported by the pretest mean score of knowledge of 44.41, which increased to 72.92 in the posttest after using the NEPWA media, and there was a significant increase in the score after using NEPWA media (0.449). In conclusion, there is a significant change in the score of knowledge before and after using the NEPWA media. This indicates that the use of the NEPWA media affects students’ knowledge. In other words, the NEPWA media can increase knowledge.


    Summary

    The variables of knowledge, satisfaction with the learning processes, and system usability in the NEPWA media showed an increase in differences and acceptance by users. The NEPWA media among student users can be said to have good system usability with a range of acceptance to be acceptable by the users. Next, in terms of the variable of satisfaction with Gagne’s model of instructional design, the majority of students were satisfied. The variable of student knowledge then increased significantly compared with the conditions before using the NEPWA media.

    Students’ prior knowledge is an adequate key so that students’ interest in the learning process can be achieved properly. This should be supported by adequate learning media, for example, the use of web-based learning media in the form of NEPWA media as supporting information media to improve cognitive learning outcomes. Learning media in the form of e-learning is an innovative app to support the teaching and learning activities of both lecturers and students [28], of which the interactive functions shall be improved to accommodate the needs of students [29] as well as the development of the NEPWA media using Gagne’s model of instructional design to support active learning and provide flexibility in improving the quality of student classroom learning by creating creative media content [30].

    To allow for the materials to be well delivered to students, a blended learning method between web-based learning methods using app or web media combined with appropriate teaching methods (face-to-face) was used, where blended learning can be used to facilitate communication and improve student learning outcomes [29]. In this case, web-based media use NEPWA media as supporting information media. In line with an earlier study [31], students can feel the benefits of implementing web-based interventions in learning, but they are not yet fully prepared to leave the traditional learning methods, including face-to-face lectures. Consistent with some previous studies [32,33], electronic-based learning provides flexible teaching methods, but both electronic-based learning and traditional teaching methods should be used simultaneously to create a superior learning style, especially for nursing courses. The explanation of the variables in this study is as follows:

    Increase in Student Knowledge

    Many previous studies have shown that web-based learning media using the blended learning approach in health profession education, in general, could significantly improve learning outcomes in terms of students’ cognitive level compared with traditional learning methods. This is based on the fact that web-based learning can optimize learning media by combining images, texts, and video, among others. The media displayed on the NEPWA media are in the form of PowerPoint slides, learning material resumes, interactive quizzes, and additional infographics, which cannot be displayed on print media. Infographics are visualization tools that can be used to improve understanding and attractiveness [34].

    This is consistent with the Cognitive Theory of Multimedia Learning [35-38], where multimedia displays can be in the form of text, images, and sound. In line with this theory, there were similar findings in another study [34] where students felt that the summary infographics provided by lecturers are very interesting and useful in understanding the topics being taught. This is in accordance with the Cognitive Load Theory [39-41], which states that humans have limited memory capacity; thus, the materials being learned should be summarized according to the topic of learning. The results of a previous study [42] showed that there are no statistical differences between the groups of students learning via a web-based app or learning in face-to-face sessions in terms of knowledge, although the web-based app media is superior to face to face; because web-based app media is able to optimize learning media with a combination of pictures, writing, and videos, [43] and the effects of learning methods on learning outcomes are very situational [44].

    Satisfaction With Gagne’s Model of Instructional Design in Learning Media

    The use of Gagne's model of instructional design in the learning process using the NEPWA media was well implemented and obtained satisfactory scores on several statement items. This is based on the fact that the first highest mean score was achieved by statement 5, “I am satisfied with the contents of the modules or materials available in this teaching media to help me study”; the second highest mean score was achieved by statement 7, “I am satisfied with brief summaries on PowerPoint slides in this instructional media”; and the third highest mean score was achieved by statement 12, “I am satisfied that I can take examination using this instructional media.” Therefore, it is expected that the high level of satisfaction with Gagne’s model of instructional design in the NEPWA media can support student learning activities and help them achieve maximum competencies.

    Many previous studies have also shown that users of web-based learning media using a blended learning approach in health professions education, in general, are satisfied with the teaching methods. In line with another study [32], when students have inadequate skills to use web-based learning media, it influences how they communicate effectively. However, the groups of students who used web-based learning media were satisfied with the program as a teaching method. The findings of another study [45] showed that positive results from web-based teaching and learning practices can increase student satisfaction and retention (the ability to store and memorize learning materials). Nonetheless, there are many challenges in performing the abovementioned activities, namely, the need to improve infrastructure and institutional support (training for educators to enhance their information technology competencies and support for multidisciplinary teams to improve web-based learning design) [45].

    SUS

    The NEPWA media has been well implemented. The results of the SUS were based on the highest mean score in statement 5, “I feel that the features of this system are working properly,” and the next highest mean score was in statement 3, “I feel that this system is easy to use.” In this way, it is expected that this innovative NEPWA media can support the teaching and learning activities of both lecturers and students and have a significant positive effect to continue and enjoy its uses because its features are working properly. However, the sustainable use of the NEPWA media is strongly influenced by the positive and subjective experiences of students, such as being confident and happy when learning by using the NEPWA media to increase learning productivity [28].

    The result of the SUS score developed in the NEPWA media was 72.24, indicating that the NEPWA media has good usability with scale C, with a range of acceptance that is acceptable to users [22,25]. This is in line with some previous studies [21,22,25], where systems or products with SUS scores >68 are considered to have a good usability category; thus, the NEPWA media has good usability. As mentioned in another study [46], the SUS score of the Moodle-based m-learning management system named Student Centered e-Learning Environment was 71.25; hence, this app can be categorized as having good usability. The fact that the NEPWA media obtained a good SUS score proved that this media is very easy to use and acceptable by student users. In fact, some of the factors that influence the SUS score are user experience, in which user groups who have repeatedly used it previously tend to give a higher SUS score compared with the group of new users [47]. Next, easy installation is crucial as users are more satisfied when an app is easily installed on a smart phone without having to download first [48].

    The study had several limitations. For example, there were no features for the statistical calculation of scores based on answers to questions, the learning materials, and questions on the NEPWA media had not been validated, and there was no repeated measurement of knowledge to determine its impact on learning outcomes.

    Conclusions

    Web-based learning media called the NEPWA media is designed to increase student knowledge. This research showed that the NEPWA media is acceptable to users, useful, has good usability, and offers flexible learning for students.

    With the potential to increase knowledge and have a good usability score, the NEPWA media can be used as supporting information media when implementing blended learning combined with Gagne’s model of instructional design with a good level of satisfaction with the learning processes. In other words, the NEPWA media can be used as alternative learning media, in addition to print media, which can help learning processes.

    Recommendations

    Future researchers are expected to conduct a validity test and add qualitative research data to support quantitative findings, which can be used to examine repetition of learning outcomes and knowledge retention. In the future, the development of the NEPWA media should focus on adding statistical calculation features to determine the number of answers per multiple-choice question examination results from users and enhancing the interactive skills of the features to be used in various fields, particularly nursing.

    Acknowledgments

    The authors wish to thank Yoyo Suhoyo, who reviewed and provided feedback related to the development of research methods and techniques for conducting the research and development; Lutfan Lazuardi, who provided feedback related to the selection of web-based learning media; Uki Noviana, who reviewed and provided feedback related to research implementation reports and writing techniques; Arif Widodo, who helped facilitate lectures and provided feedback related to the education and health promotion module; Faizah Betty Rahayuningsih, who helped facilitate the selection of courses following the curriculum of the Study Program of Nursing Science; Arum Pratiwi, the head of the Study Program of Nursing Science, who gave a research permit; Hardiansah, Director of PT Lauwba, who helped make the program website and Android app learning NEPWA media; I'ana Aulia Andari, who helped review the road of research in the field; and Muhamad Reza Utama, who helped review the way research was conducted in the field and test the NEPWA media. The researchers used independent funds in this study.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Questionnaires on satisfaction with Gagne’s Model.

    DOCX File , 18 KB
    ‎
    Multimedia Appendix 2

    Questionnaires related to the System Usability Scale.

    DOCX File , 14 KB
    1. John AD, Ronald MH. A Practical Guide for Medical Teachers. Fourth Edition. London, UK: Churchill Livingstone/Elsevier; 2013.
    2. Kominfo. Survei Indikator TIK untuk Rumah Tangga dan Individu, Balitbang SDM. Jakarta: Kementerian Komunikasi dan Informatika Republik Indonesia; 2016 Feb 18.   URL: https:/​/balitbangsdm.​kominfo.go.id/​downloads/​9/​20160218112348-Release%20infografis%20TIK%202015%20Puslitbang%20PPI.​pdf
    3. Kim S, Shin H, Lee J, Kang S, Bartlett R. A smartphone application to educate undergraduate nursing students about providing care for infant airway obstruction. Nurse Educ Today 2017 Jan;48:145-152 [FREE Full text] [CrossRef] [Medline]
    4. Ellaway R, Masters K. AMEE guide 32: e-Learning in medical education part 1: learning, teaching and assessment. Med Teach 2008 Jun;30(5):455-473. [CrossRef] [Medline]
    5. Tan T, Liu T, Chang C. Development and evaluation of an RFID-based ubiquitous learning environment for outdoor learning. Interact Learn 2007 Dec;15(3):253-269. [CrossRef]
    6. McMullan M. Evaluation of a medication calculation mobile app using a cognitive load instructional design. Int J Med Inform 2018 Oct;118:72-77. [CrossRef] [Medline]
    7. Gongjun Y, Danda BR, Hui S, Awny A. Developing and applying smartphone apps in online courses. J Inf Syst Educ 2014;25(2):12 [FREE Full text]
    8. Zakaria N, Jamal A, Bisht S, Koppel C. Embedding a learning management system into an undergraduate medical informatics course in Saudi Arabia: lessons learned. Medicine 2.0 2013 Nov 27;2(2):e13 [FREE Full text] [CrossRef] [Medline]
    9. Lu'mu L. Learning media of applications design based android mobile smartphone. Int J Appl Eng Res 2017;12:6576-6585 [FREE Full text]
    10. Bidin S, Ziden AA. Adoption and application of mobile learning in the education industry. Procedia Soc Behav Sci 2013 Oct;90:720-729 [FREE Full text] [CrossRef]
    11. Briz-Ponce L, Pereira A, Carvalho L, Juanes-Méndez JA, García-Peñalvo FJ. Learning with mobile technologies – students’ behavior. Comput Hum Behav 2017 Jul;72:612-620 [FREE Full text] [CrossRef]
    12. Tang B, Coret A, Qureshi A, Barron H, Ayala AP, Law M. Online lectures in undergraduate medical education: scoping review. JMIR Medical Education 2018 Apr 10;4(1):e11 [FREE Full text] [CrossRef] [Medline]
    13. Fandianta F, Sanjaya GY, Widyandana D. Meningkatkan pengetahuan mahasiswa dengan memberikan fleksibilitas belajar mengajar melalui metode blended learning. Med J Indones 2013 Jun;2(2):146-153 [FREE Full text]
    14. Khadjooi K, Rostami K, Ishaq S. How to use Gagne's model of instructional design in teaching psychomotor skills. Gastroenterol Hepatol Bed Bench 2011;4(3):116-119 [FREE Full text] [Medline]
    15. Gagne RM, Wager WW, Golas KC, Keller JM, Russell JD. Principles of instructional design, 5th edition. Perform Improv 2005 Feb;44(2):44-46 [FREE Full text] [CrossRef]
    16. Sugiyono D. Metode Penelitian Kuantitatif. Bandung, Indonesia: Alfabeta; 2016.
    17. Branch RM. Instructional Design: The ADDIE Approach. New York, USA: Springer; 2009.
    18. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage Publications; 2014.
    19. Notoatmojo S. Metodologi Penelitian Kesehatan. Jakarta, Indonesia: Rineka Cipta; 2018.
    20. Buzzetto-More AN. Student perceptions of various e-learning components. Interdiscip J e-Skills Lifelong Learn 2008;4:113-135 [FREE Full text] [CrossRef]
    21. Sharfina Z, Santoso HB. An Indonesian Adaptation of the System Usability Scale (SUS). In: 2016 International Conference on Advanced Computer Science and Information Systems. Malang: IEEE; 2016 Presented at: ICACSIS'16; October 15-16, 2016; Malang, Indonesia p. 145-148   URL: https://ieeexplore.ieee.org/document/7872776/metrics#metrics [CrossRef]
    22. Mohamad MF, Yaacob NA, Yaacob NM. Translation, cross-cultural adaptation, and validation of the malay version of the system usability scale questionnaire for the assessment of mobile apps. JMIR Human Factors 2018 May 14;5(2):e10308 [FREE Full text] [CrossRef] [Medline]
    23. Sastroasmoro S. Dasar-Dasar Metodologi Penelitian Klinis. Jakarta, Indonesia: Sagung Seto; 2018.
    24. Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors. Usability Evaluation In Industry. New York, USA: CRC Press; 1996:207-212.
    25. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. Usab Profess Assoc 2009 May;4(3):114-123 [FREE Full text]
    26. Brooke J. SUS: a retrospective. J Usability Stud 2013 Feb;8(2):29-40 [FREE Full text]
    27. Nursing Education Progressive Web Application.   URL: http://nepwamobile.com/adm/login [accessed 2020-09-02]
    28. Wang L, Lew S, Lau S, Leow M. Usability factors predicting continuance of intention to use cloud e-learning application. Heliyon 2019 Jun;5(6):e01788 [FREE Full text] [CrossRef] [Medline]
    29. Shorey S, Kowitlawakul Y, Devi MK, Chen H, Soong SK, Ang E. Blended learning pedagogy designed for communication module among undergraduate nursing students: a quasi-experimental study. Nurse Educ Today 2018 Feb;61:120-126 [FREE Full text] [CrossRef] [Medline]
    30. Leow FT. Interactive multimedia learning: innovating classroom education in a Malaysian university. Turkish Online J Educ Technol 2014 Apr;13(2):12 [FREE Full text]
    31. Koch J, Andrew S, Salamonson Y, Everett B, Davidson PM. Nursing students' perception of a web-based intervention to support learning. Nurse Educ Today 2010 Aug;30(6):584-590 [FREE Full text] [CrossRef] [Medline]
    32. Abdelaziz M, Kamel SS, Karam O, Abdelrahman A. Evaluation of e-learning program versus traditional lecture instruction for undergraduate nursing students in a faculty of nursing. Teach Learn Nurs 2011 Apr;6(2):50-58 [FREE Full text] [CrossRef]
    33. McDonald EW, Boulton JL, Davis JL. E-learning and nursing assessment skills and knowledge - an integrative review. Nurse Educ Today 2018 Jul;66:166-174 [FREE Full text] [CrossRef] [Medline]
    34. Gallagher SE, O’Dulain M, O’Mahony N, Kehoe C, McCarthy F, Morgan G. Instructor-provided summary infographics to support online learning. Educ Media Int 2017 Aug 28;54(2):129-147 [FREE Full text] [CrossRef]
    35. Azimi E, Mousavipour S. The effects of educational multimedia in dictation and its role in improving dysgraphia in students with dictation difficulty. Contemp Educ Technol 2014;5(4):331-340 [FREE Full text] [CrossRef]
    36. Mayer RE. Cognitive theory of multimedia learning. In: The Cambridge Handbook of Multimedia Learning. Cambridge, USA: Cambridge University Press; 2005:31-48.
    37. Mayer RE. Multimedia learning. In: Psychology of Learning and Motivation. New York, USA: Elsevier; 2002.
    38. Mayer RE. Multimedia Learning. Second Edition. Cambridge, USA: Cambridge University Press; 2009.
    39. Gardner K, Aleksejuniene J. PowerPoint and learning theories: reaching out to the millennials. Teach Learn J 2011;5(1):1-11 [FREE Full text]
    40. Ormrod JE. Human learning Sixth Edition. In: Human Learning. Sixth Edition. London, UK: Pearson; 2012.
    41. Young JQ, Van Merrienboer J, Durning S, Ten Cate O. Cognitive load theory: implications for medical education: AMEE guide no 86. Med Teach 2014 May;36(5):371-384 [FREE Full text] [CrossRef] [Medline]
    42. Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud 2014 Jan;51(1):136-149 [FREE Full text] [CrossRef] [Medline]
    43. Öztürk D, Dinç L. Effect of web-based education on nursing students' urinary catheterization knowledge and skills. Nurse Educ Today 2014 May;34(5):802-808 [FREE Full text] [CrossRef] [Medline]
    44. Voutilainen A, Saaranen T, Sormunen M. Conventional vs e-learning in nursing education: a systematic review and meta-analysis. Nurse Educ Today 2017 Mar;50:97-103 [FREE Full text] [CrossRef] [Medline]
    45. van Rensburg ES. Effective online teaching and learning practices for undergraduate health sciences students: an integrative review. Int J Afr Nurs Sci 2018;9:73-80 [FREE Full text] [CrossRef]
    46. Banimahendra RD, Santoso HB. Implementation and evaluation of LMS mobile application: scele mobile based on user-centered design. J Phys: Conf Ser 2018 Mar 16;978:012024 [FREE Full text] [CrossRef]
    47. McLellan S, Muddimer A, Peres SC. The effect of experience on system usability scale ratings. J Usability Stud 2012;7(2):13 [FREE Full text]
    48. Erevelles S, Srinivasan S, Rangel S. Consumer satisfaction for internet service providers: an analysis of underlying processes. Inf Technol Manag 2003;4(1):69-89 [FREE Full text]

    ‎
    e-learning: electronic learning
    m-learning: mobile learning
    NEPWA: Nursing Progressive Web Application
    SUS: System Usability Scale

    Edited by E Borycki; submitted 01.05.20; peer-reviewed by A McLean, H Akram; comments to author 20.05.20; revised version received 02.07.20; accepted 28.07.20; published 13.11.20

    Copyright

    ©Deny Yuliawan, Doni Widyandana, Rachmadya Nur Hidayah. Originally published in JMIR Nursing Informatics (https://nursing.jmir.org), 13.11.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.