Developing a Course for Learning at Scale in Moodle : A Design-Case Based on Learner ' s Experience at a Private University in Malaysia

To be relevant to the 21 st Century Learner’s needs, Taylor’s University (TU), a private university in Malaysia, revamped its curriculum to be learner-centred. The Malaysian Studies 3 (MS3) course aimed to be the exemplar module site for future module site development for learning at scale (L@S). The learners are undergraduate students, taking MS3 as an elective. In the two design experiments, the learner’s experience was examined using a focus group after undergoing a 14-weeks of L@S entirely online in Taylor’s Integrated Moodle eLearning System (TIMeS). This design-case seeks to investigate the learner’s experience of (1) usability and (2) learning. Two prototypes of MS3 were developed and tested. For the tryouts, Cohort 1 as pre-test and Cohort 2 as post-test after implementing the LDS framework model as a tool to guide learning design in the design refinement. In conclusion, the findings showed ‘teacher presence’, and ‘technical issues’ are the two major themes that influence learners’ positive experience. To ensure positive learning experience, course instructors must ensure engagement with learners is purposefully designed. The study contributes to design based implementation research (DBIR) under the umbrella of design and development research (DDR), and the practical problem in a module development project for L@S.


I. INTRODUCTION
The age of transformative technology formed by the 4th Industrial Revolution (4IR) has changed the landscape of higher education at lightning speed. Change of learning mode to be conducted entirely online for learning at scale (L@S) is no longer an impossible feat. Today's learners live in a world where knowledge is accessible at their fingertips using smart devices. The 21st-Century disruptive learning technologies allow learners to interact and engage with their learning material, course instructors, and peers as long as they have an internet connection.
The higher education institutions in the developed nation paved the way for creative approaches to disruptive Manuscript  education delivery. The Malaysian Higher Education Institutions (HEI) on the other hand, need to catch up to ensure they are delivering education that is relevant to the needs of today's learners and globally accepted. Taylor's University (TU) is one of the private universities in Malaysia that has made great strides in taking the lead on an innovative approach to learning and teaching. One of the ways was to reform its curriculum delivery and learning methods to the new Taylor"s Curriculum Framework (TCF), allowing learners to curate their learning paths [1]. Part of this educational reformation will also include changing the approach in how each course is delivered, and this includes delivering learning entirely online for L@S.
This research aims to experiment in a small scale, with the Malaysian Studies 3 (MS3) module site as a tryout to compare the first year learner"s experience after undergoing a 14-weeks of learning entire online for L@S. A comparison between Cohort 1 as a pre-test and Cohort 2 post-test after the intervention was implemented. The intervention is a learning design strategy framework (LDS Framework) model used as a tool to aid the learning design strategy for the practitioners when working collaboratively to refinement the design and development of the MS3 module site. The main contribution of the study is both theoretical and practical. Theoretically, the study contributes to the design based implementation research (DBIR) under the umbrella of design and development research (DDR) approach and the problem in practice in an actual module site development project in Moodle for L@S in a Malaysian private university setting.

A. Background
The transformative shift in how education can be delivered under the new TCF in TU gave promise for learning that can be personalised and ubiquitous [1], [2]. A new method of education delivery also means a change from how the learning is conducted, significantly if its mode of learning has shifted to entirely online. This education transformation under the new TCF caused gaps in practice in how learning should be delivered according to the TCF that called for constructive alignment of outcome-based learning, graduate capabilities, and assessment. Further, educators in TU do not necessarily come from an education background due to its practice in hiring experts from the industry. Hence the practice of constructive alignment and learning design can be a challenge [3]. In ensuring the practitioners are properly guided, a learning design strategy framework (LDS Framework) was proposed by the researcher to be utilised as a tool to guide practitioner"s learning design strategy to ensure they are guided when developing a module site for L@S [4]- [6].
Malaysian Studies 3 (MS3) is the first course to be transformed entirely online from its traditional lecture theatre mode into fully online mode. In this pilot project, learning is to be delivered to a massive number of TU"s learners using Taylor's Integrated Moodle eLearning Framework (TIMeS). This project utilises TIMeS as a platform for L@S for the first time. Prior to this project, TIMeS has never been tested for its ability to offer learning at scale (L@S). Hence MS3 is the test module site to be used as the exemplar module site for more courses to be developed for L@S and conducted entirely online. In order to test the effectiveness of a systematic learning design strategy using the LDS Framework implemented in the MS3 module site development and refinement, a focus group study was conducted with learners from Cohort 1 as a pre-test and Cohort 2 after the treatment was applied as post-test. This study aims to answer the following research questions: 1) What is the student"s experience of the MS3 module site for L@S? (Developed before the LDS Framework model was implemented) 2) What is the student experience of L@S based on the MS3 module site? (Developed after the LDS Framework model was implemented) The study on the MS3 pilot project to transform L@S from face-to-face (F2F) to entirely online in TU, is a complex and multi-dimensional project. Previously the author had published on its research approach, and different research phases [4]- [9].

B. Learning at Scale (L@S)
The Massive Open Online Course (MOOC) is an example of a mode of learning and teaching that facilitates a large-scale module and is open to all. Although MOOC can scale up and democratize education, this teaching method was strongly criticised as challenging the well-established research on online education [10], [11]. To date, many MOOC learning management providers such as Coursera, edX, Udemy and FutureLearn, have modified their business strategy to offer instead micro-credentialing courses that allow more flexibility to learners and ability to monetize the courses offered for L@S, [12]. Therefore, what we know was right about MOOC or micro-credential (MC) courses is that they can democratize and deliver life-long education [13]. Nevertheless, it is essential to reassess the capacity of higher learning institutions (HEI) such as TU, to endorse courses for L@S.
We come to understand "learning at scale" or abbreviated as "L@S" became a field of research since 2014, inspired by MOOC. L@S research brings together two research communities among the learning scientists and "computer and data scientists" for improving learning happening at massive scale. In its application today, L@S expanded to a broader term to include various types of L@S happening entirely online. "L@S is where learning is massively undertaken as private courses, micro-credentials, or in a community of practice such as the site DeviantArt where participants collectively review and critique participant"s work allowing learning and creation of new knowledge [14]. New learning platforms such as intelligent tutoring system using mobile app occurs at a high student per teacher ratio" [14].
In determining the number of learners to be considered as L@S in the online setting, massive was described as "preferably more than one thousand students, while more than one hundred for face-to-face setup" [15]. In TU"s private university context, the researcher has redefined L@S as having at least one hundred students per student-instructor ratio, considering a private university like TU with a low student ratio per classroom. TU realizes providing L@S has its advantages, opening new possibilities, yet the challenges of conducting courses for L@S must be fully understood by their practitioners [16], [17].

C. A Learning Design Framework for Course Development
Past research informed on the importance of having a framework that addresses how to systematically design an online course [18]- [21]. A published study on learning design informed that a framework could become a tool to guide inexperience course designers [22]. Novices tend to think in abstraction and face challenges in transforming pedagogical knowledge into practical design features. The course instructors, on the other hand, faced challenges in shifting their thinking from the educators to the "designers' and having to put in some long hours in planning for the learning design of an online course [23]. Indeed, in revisiting the MS3 course for design refinement, the researcher worked with the practitioners to ensure real planning of the learning design strategy was being done in the LDS Framework tool before the development to improve MS3 module site.

D. Moodle LMS for L@S
With many advancements and developments in the Learning Management System (LMS), for Education 4.0, Moodle is still relevant today as an LMS. Moodle stands for "Modular Object-Oriented Dynamic Learning Environment". Many learning resources for interactive learning can be used in Moodle. This includes a peer-to-peer assessment or group-to-group assessment, Wiki for collaborative learning, immersive videos and virtual reality user experiences using H5P and the Moodle mobile app. Digital badge and level up can be used to motivate learners in the online learning environment [24], [25]. TU adopted Moodle as the LMS and rebranded as Taylor"s Integrated Moodle e-Learning System (TIMeS). Since it is open-source, it is a more affordable option for a learning institution [26].
Although TIMeS was utilised since 2011 as TU's LMS, its finesse in performing courses for L@ has never been tested. The MS3 module site will be the first test site for a course in the University for L@S in the TIMeS. Inside TIMeS, learners are supposed to be able to learn through the content and e-activities and engage with their course instructors and peers. The Wiki space is being used for group discussion and reporting. On the other hand, the Workshop activity allows learners to assess their peers. Engaging with learners and course instructors can be done using Forum and an external tool Telegram app for rapid communication and dissemination of information.
This study will evaluate learners" experience of learning MS3 course in TIMeS before any intervention using the learning design strategy framework (LDS Framework). A design experiment was conducted as "tryout" to compare learners" experience before and after the improvement of the learning design of MS3 was done using the LDS Framework.

E. Design-Based Implementation Research as the Research Approach
In its entirety, this study employs the design-based implementation research (DBIR), a research approach under the umbrella of design and development research (DDR). This study differentiates itself from other DDR because it included implementation into the research design proses because the concern of the researcher on developing capacity for sustaining change in the systems of design and development of module site for L@S in TIMeS. Hence, in the execution of the operations of the module site in development for this research, the testing of the innovation was conducted to improve teaching and learning in its implementation [27]. The bespoke intervention is the LDS Framework used as a tool for learning design. However, the best method to measure the effectiveness of the LDS Framework is by measuring learner's experience of learning in MS3, which was designed using the LDS Framework.
McKenney and Reeves [28] proposed the intervention used for educational design research to be used as tryouts, "to study how the intervention works, what participant think or feel about them, and the results they yield. Tryouts take place when (a prototype of) the intervention is field-tested in a natural setting" (p. 176). A study by De Vries, Boersma, and Pieters, [29] on using e-mail as a tool for online reflection also used method of pre-test and post-test of 2 different sets of students as a tryout to understand the intervention using e-mail as a tool from a design perspective to aid collective reflection. Here, before and after the LDS Framework prototype was utilised in practice, the pre and post-test seek to measure evidence of an effect to the learner's experience using focus group.

III. RESEARCH METHODOLOGY
In a design-based study, a focus group may be undertaken at the earlier stages of intervention as an opportunity to assess and revise research instrument to be used later; or as a pre/ post-test measures to seek proof of an impact with the intention to "explain how and why the effect is observed (or not)", p. 176 [28]). Procee treated the focus group as one of the reflection strategies, for example, to analyze the interactions of pupils and teachers with the new learning environment [30]. In the discovery process of design-based research, a focus group can be used as a way of evaluation in proposing design changes [31].
A focus group study allows in-depth examination on learner"s experience. The focus group methodology originated from marketing and health traditions [32]. However, the focus group is used for decades as a tool by social scientists; for example, in sociology [33], [34] in architecture [35] and research in curriculum design [36]- [38]. The most common aim of a focus group interview is to discuss a topic in-depth when it is appropriate to understand the issues being studied [39]. Therefore, a focus group using semi-structured interview was conducted with both learners from Cohort 1 and Cohort 2 after they have undertaken 14-weeks of MS3 course entirely online in TIMeS.
In this study, the focus group is useful to be used as a tool for evaluation in the research phase, as an exploration of the learner's experience of learning MS3 entirely online for L@S. Learners from Cohort 1 may provide some insights useful for enhancing the learning design strategies for Cohort 2. The approach suggested by previous studies are five participants to be the minimum number in a focus group [36], [39]. The aid of their course instructors was sought to find the volunteers and consent was obtained from TU"s Board of Ethics before the study was conducted. An interview protocol was prepared, and consent was obtained from the participants before the start of the focus group session.
This study utilised the agile Successive Approximation Model 1 (SAM1) as its research design to operationalize the implementation of course rollout rapidly [40]. SAM1 is a simple, agile, iterative method that begins with evaluation and ends with evaluation (See Fig. 1). Evaluation of learner"s experience was conducted at the start of the research (pre-test) and the end of the research phase (post-test). It is important to note that the SAM1 agile model allows for the MS3 module site, to be developed as a minimal viable product (MVP) so that the product can be tested by real users and obtain the necessary feedback needed for design improvement in the subsequent iterations. A thematic data analysis approach was carried out using the Six Steps of Thematic Analysis [41]. The steps include 1) Familiarization of data, 2) Initial coding, 3) Generating themes, 4) Validating of reliability themes, 5) Defining and naming themes and 6) Interpretation and reporting.
The responses were divided into two groups: 1) positive, and 2) negative. Guskey"s 5 Levels of Data was adapted by the researcher in categorizing and coding the themes from the interview transcripts [42].  Guskey and Kirkpatrick"s instrument must be implemented one level at a time. This study takes into consideration data at Level 1 and Level 2 that best describes learner"s experience [43]. Hence, the main themes focused on the learner"s experience of L@S the 1) "usability" of the MS3 module site conducted in TIMeS and 2) how they experienced "learning" the MS3 entirely online in TIMeS. The results and findings of the learner"s experience were shared with the practitioners as a design review process to improve the next iteration of the MS3 exemplar module site. Further, sharing the findings with the practitioners acted as a member's check of the focus group's data validity and transferability.

IV. THE DESIGN EXPERIMENT
Two design experiments were conducted as pre-test and post-test to compare the experience of learners from Cohort 1 and Cohort 2 (after implementing the LDS Framework).

A. The First Design Experiment
Learners of Cohort 1 are learners who had undertaken the MS3 course in TIMeS before any LDS Framework was utilized during MS3 course development.

1) Participant
A total of 11 out of 193 students from Cohort 1 volunteered for the pre-test. The focus group interview with Cohort 1 learners was performed to provide an opportunity to gather more concrete data as a pre-test (Patton, 1990;Liamputtong, 2015) to understand the impact of systematic learning design. For better preparation of potential iterations of MS3 module site, the results from the pre-test were shared with the practitioners. The 11 participants were coded from MS32018C101 to MS32018C111, a bracket (M) for example, MS32018C101(M) stands for male participants, while MS32018C102(F) stands for female participants for Cohort 1.

2) Practical considerations
In preparation for the MS3 module site pre-test, the course developers had a short development timeline, due to the course instructors and subject matter experts (SME) were not able to give their full commitment due to their workloads. This caused delays in the tasks of developing the content. Hence, for the alpha version of MS3 module site, the course content was developed using the MS3 course"s Scheme of Work (SOW) and no systematic and collaborative learning design planning was done by the practitioners. To ensure the timeline is met, most of the design and development of MS3 was done by the e-Content Development Specialist without considering whether the course instructors were able to carry out the activities on their own. Development of course content was focused on instructional resources such as videos and enhanced PowerPoint slides. However, it was only partially developed; for example, only four new instructional videos and four new slides were developed. The learners were expected to do their reading from the textbook and non-enhanced PowerPoint slides prepared by the SMEs. Learning activities, on the other hand, were developed by the e-Content Development Specialist such as the collaborative activity Wiki for group work discussion, and Workshop for peer to peer assessment (See Fig. 3). The MS3 was developed with an assumption the course instructor is the SME of their content and know how best to teach the course. Thus, the main focus of the e-Content Development Specialist leading the course development was to ensure the MS3 module site was ready with the MVP content for rollout when the semester starts. The interface design used the "collapse topic" course format (Fig. 2). The course instructors were given some training and coaching on managing the activities and resources in TIMeS just before the new semester started.  Although developed as MVP, a learner-centric design approach was the primary consideration and TIMeS tools that allowed for self-directed learning such as a progress bar and digital badges were implemented. The Progress Bar served to alert learners to upcoming assignments and to inform the completion of activities. Digital badges, on the other hand, told learners of their achievement by rewarding the success of task completion or reinforcing good behaviour.
The course instructors were expected to be familiar with Wiki used for collaborative learning assignment submission, but new tools such as Workshop an activity tool, and rating scale a new function used for peer-to-peer assessment were newly installed. These new tools were added with the purpose to assist the course instructor to manage a course for L@S. Restriction settings were also new functions that were enabled to control learner"s pace for self-directed learning. By using restrictions, learner"s access to new sections or activities is controlled by a date or score restriction.
The Workshop is a peer-to-peer or group-to-group evaluation method where the learner or group of learners may assess the assignment of their peers based on a rubric shared by the course instructor. The Workshop activity enables the course instructor to appoint learners as reviewers to one or more assignments of their peers. When completed, the task and the act of evaluation will each bear a particular assessment mark. The course instructor may also mark the assignment of the student and become as one of the reviewers (See Fig. 4). For the first design experiment, the e-Content Development Specialist assumed the course instructor know how to engage with their learners and did not put define the engagement activities in the learning design considerations. At this point, the course instructor was told they could use Forum, and the app Telegram was used as a form of rapid response for course instructors to communicate with their massive number of learners.

3) Procedure
A range of data was gathered to gain insight into the course instructor"s implementation of MS3 module site for L@S, and the learner"s experience of "usability" and "learning". An open-ended interview using a focus group of the 11 participants was gathered to gain the learner"s insight. Proper approval from the board of ethics from TU was obtained to conduct the research. An interview protocol was observed where the participant"s consent was obtained before the interview. A co-researcher was present as a member check to establish the validity and credibility of the session. Notes taken by the researcher was compared with the co-researcher by the end of the session. A transcriber was appointed to transcribe the interview. The transcription was later coded to themes using Atlas.ti version 8 tool to assist the thematic analysis process.

B. The Second Design Experiment
Learners of Cohort 2, on the other hand, are learners who had undertaken the MS3 after an improvement was made to the MS3 module site in the subsequent semester.

1) Participant
A total of five out of 100 students from Cohort 2 volunteered. For Cohort 2, it was a challenge to find volunteers since the semester break had just started, and most students had dispersed from the campus. The learning design strategy of MS3 was improved using the LDS Framework as a tool for learning design. For Cohort 2, the five participants were coded according to their course code (MS3), preceded by the year (2019), cohort number (C2) and the student number, e.g., number (01) as MS32019C201. They were therefore encoded from MS32019C201 to MS32019C205, MS32019C201(M) for male participants, and MS32019C202(F) for a female participant.

2) Practical considerations
By no means is the MS3 second iteration is the final, perfect version. The feedback from learners can be used to iterate any stage of the learning design strategy for the next design improvement. The second design experiment, i.e., the post-test will be able to provide some feedback on usage of the LDS Framework as a tool to plan for learning design to develop/improve a module site for L@S that is aligned with the new TCF, conducted entirely online. The practitioners adhere to the following nine principles of learning design of the LDS Framework: 1) Course structure 2) Module learning outcome(s) 3) Taylor"s Graduate Capability(s) 4) Assessment 5) Topic(s) according to Sequence 6) Taylor"s Pedagogy(s) 7) Type of Learning and Teaching Activities 8) Type of Resource(s) 9) Digital Badge(s) For the second design experiment, the focus is on personalized yet participatory learning, as learners learn how to collaborate with their peers in TIMeS. Similar to Design Experiment 1, restriction access settings were applied, but instead of restricting learner"s access based on date or marks, the restriction was set for activity submission and completion. By designing the learning design in this manner, each learner was still able to personalize the pace of their learning without being too restrictive.
The interface design was further simplified for a clean and easy to use approach, using "button course" format to eliminate the module site from becoming too scrolly. The planning in the LDS Framework further broke down each section to a smaller chunk to reduce the scrolly effect.
Increasing interaction between instructor-learners and their peers was the critical aim when planning and restructuring the learning design strategy. In order to increase 'teacher presence', more synchronous online session using YouTube Live was conducted and repeating the presence of the course instructor, when they post short videos to summary learning takeaway. The course instructor was also encouraged to make weekly announcements, reacting, and replying to learner"s posts and questions promptly. The Telegram Messaging Tool was again retained as the Social Media App of the Course because of its ability to handle a large group of learners.
In the meantime, the design of the Forum learning object was set to function not just to engage with learners. Using Forum, learners can also rate their peer"s responses using a rating scale. When learners know peers will assess their response, this motivated them to submit quality responses in the Forum.
The problematic Wiki was replaced with OU Wiki activity. OU Wiki is a learning object that can also be used for collaborative activity, yet the WYSIWYG editor made the learner able to carry out their posting easily. Further, the course instructor was able to annotate and then share out exemplar group project submission with other groups. Video guides were also added to accompany the instruction on how to conduct the OU Wiki and Workshop activities to guide learners for self-paced learning.
With the follow-up coaching sessions were given to the course instructor, they were able to confidently guide their learners in both activities that utilize OU Wiki and Workshop. The scaffolding given to the course instructors enabled to minimize technical errors when the course instructors edited the setup of the learning objects. For the second design experiment, the practitioners decided not only to add new resources but to also further edits on the videos created and used for Design Experiment 1. Besides improvement on the interactivity of the instructional videos, the videos were also chunked into a small digestible byte of learning objects not to exceed 10 minutes per video.

3) Procedure
Just as in the Design Experiment 1, a range of data was gathered to gain insight of the course instructor"s implementation of MS3 module site for L@S and the learner"s experience of usability and learning. An open-ended interview using a focus group of the five participants was gathered to gain the learner"s insight. An interview protocol was observed where the participant"s consent was obtained before the interview. A co-researcher was present as a member check to establish the validity and credibility of the session. Notes taken during the session were compared with the co-researcher by the end of the session. A transcriber was appointed to transcribe the interview. The transcription was later coded to themes using Atlas.ti version 8 tool to assist the thematic analysis process.

A. Design Experiment 1 1) Learners experience of MS3 on usability in design experiment 1
Participants of focus group from Cohort 1 experienced the MS3 module site as 'useful', "good" and "beneficial" linked to central theme (1) usability because of its "flexibility" and "easy access" from TIMeS. These were the positive responses recorded from Cohort 1.
For the researcher, the negative feedback on "usability" was more disconcerting because it indicates that learners were disappointed with the experience of usability of the Some of the negative comments related to "usability" were due to the course instructor"s failure to explain during the first briefing session, how particular learning objects work, for example, Wiki and Workshop. As an example, a negative comment from learner coded as MS32018C106(M) on Wiki extracted as "I tried to make my own Wiki  In the MS3 alpha version, the focus of the interface design was to improve on its aesthetics. However, learners were unimpressed because the interface was "scrolly" and some optimization of the graphics still need to be improved because it causes lagging to the MS3 module site loading time. For example, MS32018C108(M), noticed the interface in MS3 was different. He mentioned "the interface was very different from everything else on TIMeS", but he was not impressed because he feels it was still not user friendly and "scrolly" was the word he used to describe his experience of the interface.
On the other hand, the positive comments given by participants motivated the course instructor to have confidence in teaching entirely online. The negative feedback motivates the practitioners to make further improvement on the learning design strategies in the newly developed LDS Framework.
Many of the more prevalent sub-themes include the concept of flexibility that comes with conducting a class in a completely online mode. Online mode enables multiple access to learning material using multiple devices. None of the participants from Cohort 1 focus group protested that they had to learn with their peers online. It was noted, while learners could, to some degree, be able to accept technical problems related to the formatting of learning objects, interface design, slowness, downtime maintenance, or network and infrastructure issues in TIMeS; they were unaccepting when they feel there was no teacher presence online.
Consequently, learning design strategy improvement was made in the LDS Framework tool to aid the process of the MS3 module site refinement. The practitioners tried to address and implement design solutions to address the issues that prevented positive experience related to usability and learning MS3 by Cohort 1.

B. Learners Experience of MS3 on Learning in Design Experiment 1
Learning entirely online for L@S was a new experience for learners of Cohort 1. This form of learning was seen as useful by Cohort 1 because learners can have quick access to learning resources. The forum activity was perceived to be easy to complete. However, there were a lot more negative responses given by Cohort 1 as compared with positive responses.
Of the negative responses, what is more important to highlight, is the response regarding "no teacher presence" since this relates to the course instructor and how they can use this design-case to improve on their learning design strategy. From the responses gathered, the participants from Cohort 1 perceived teacher"s presence were inadequate because the course instructor did not fully engage with the learners virtually in the class activities, post announcement timely, or remind and speak with the learners in the Telegram social media channel. Further, in Malaysia, Telegram is not mainstream social media. MS32018C107(F) complained: "it was not that effective with the online, like totally online, because didn't put out announcements and things in Telegram and I don't think every one of us uses Telegram. It's not that popular for all because mainly for the other classes I have seen they we were using WhatsApp group, and then Facebook, and TIMeS that was more effective for us". The Telegram app was decided to be used as the social media tool due to its capacity so support L@S.

C. Design Experiment 2 1) Learners experience of MS3 on usability in design experiment 2
International Journal of Information and Education Technology, Vol. 11, No. 7, July 2021 309 MS3 conducted entirely online in TIMeS. Although practitioners may have control over the learning design strategy and specific aspects of the MS3 module site design, technological problems related to the maintenance problem of TIMeS or the speed of the network were out of their control. The sub-themes related to negative responses were: 1) Technical problem in TIMeS, 2) Unacquainted with TIMeS"s tools, i.e., Wiki, Workshop, Journal, Forum, Progress Bar, Digital badges, 3) TIMeS"s interface is not user friendly.
The following sub-themes of negative responses were gathered from Cohort 1, 1) improve quality of instruction (notes and videos), 2) no teacher presence, including "poor chat response time", "no engagement", "not effective", "communication breakdown", 3) "not understanding the purpose of digital badges", 4) "peer-to-peer assessment backfires", 5) "no motivation", and 6) "cannot self-regulate". Of the themes which were perceived by the researcher to be more serious and will impact learners having a good learning experience, must be addressed to the practitioners for the next design refinements.
Though 'technical problems' and "no teacher presence' were still emerging issues of the adverse reaction, it was more situational. Only one of the participants in the focus group MS32019C203(F) encountered technical problems, while the majority of the participants in the Cohort 2 focus group did not encounter any technical difficulties. 'Flexible' and 'teacher presence' was also the recurrent themes with a new theme emerging 'peer support' in Cohort 2 positive responses. The themes will be explored in the following paragraphs.
For the positive responses, more positive sub-themes emerged related to Cohort 2 experience of "usability" besides "good" or "beneficial" such as "flexible", "relaxed", "convenient", "teacher presence", "teacher helpful", "peer-to-peer learning is good", "like", communication is easy", and "useful". The positive experience of learners of Cohort 2 related to "usability" outweighs the negative reactions. Only MS32019C202(M)"s informed "technical problem" as "Sometimes to be honest when I finish it (the activity) the box (progress box) is still not checked. Yeah, there is sometimes", which is a known issue due to the "cron job" or the systems scheduler update in TIMeS.

2) Learners experience of MS3 on learning in design experiment 2
For the main theme relevant to how participants 'experience learning' at the MS3 model site, positive learning responses include 'topic is interesting,' 'like group work,' 'teacher presence,' "teacher helpful", 'like collaborative assignment.' Response from MS32019C202(M) informed of experiencing the course instructor being helpful as "I think mostly it was clear only sometimes I don't understand, but I can just message her". There was no negative response given by participants to indicate that learning MS3 entirely online did not meet lesson objectives.
The negative response still emerged, for example, a negative response such as ""no feedback from instructor", related to an experience when the learner did not get the feedback from the course instructor as to how she expected, such as MS32019C203(F) "I got a low mark for it (a test). So, I asked her to check again, but she didn't check it". Further, when learner felt the response from the course instructor was not timely as "The disadvantage is that if I have a question, I need to make a meeting with her and sometimes she's busy, or I have class so we cannot make a meeting".
MS3201902(M) on the other hand emphasized, feeling more relaxed taking MS3 entirely online, but acknowledged there were certain topics which were harder to understand. It would be easier if it were addressed in the traditional classroom setting "the disadvantages, sometimes it's hard to understand for me. So, I need to ask the instructor". Finally, all o the learners of Cohort 2 felt that learning MS3 course entirely online fulfilled the learning outcomes.

VI. DISCUSSION AND RECOMMENDATION
In evaluating the learner's experience, findings in both pre-test and post-test shows there tend to be one recurrent pattern for the learners to provide positive responses related to 'usability' and 'learning' and that is 'teacher presence'. On the other hand, two recurrent themes emerged for learners to give negative responses, and these are 'no teacher presence' and 'technical issues'. In Cohort 1 'teacher presence' was not identified spoken in a positive light. Learners in Cohort 1 did not have a positive experience learning MS3 entirely online, although initially, they agreed that learning entirely online provides the 'flexibility' they need. 'Flexible' is the recurrent theme Cohort 1 discussed throughout the focus group study regarding its 'usability'.
A recurrent theme that kept appearing in Cohort 1 was 'technical problems'. Post interview, upon discussion with the practitioners and the eLA's Learning Technologist Team, it was found out, the key problem associated with 'technical problems' was due to the incorrect settings of the learning activities and resources made by the course instructor. At the same time in a separate study, the needs of the practitioners, i.e. the course instructors were also established that they need continuous scaffolding from the e-Content Development Specialists until they become familiar with the settings of each of the TIMeS activities and resources [7]. A repeated exposure to raise the course instructor's awareness on learning design through scaffolding using training and coaching, to adequately plan their learning design strategy, using the LDS Framework tool must be implemented as a system. Through guidance from the LDS Framework for a systematic way to develop a module site, the course instructors were able to understand the importance of planning towards managing an entirely online course to establish 'teacher presence'.
To improve the learning design strategies for the subsequent MS3 module site, the researcher called for a design-review meeting with the practitioners to review the comments from the participants of Cohort 1. The purpose of sharing the results with the course instructors and other e-Content Development Specialists was to use the MS3 exemplar module site as a design-case to enhance the delivery of L@S of upcoming module sites. Together with the practitioners, the negative remarks with regards to "usability" and "learning" were analysed, and the importance of changing the learning design strategy for Cohort 2 was taken into account into the learning design strategy in the LDS Framework tool for the improvement of the MS3 module site. During the review, comments made by learners to indicate their personal preference were ignored, such as preferring to learn from slides or not being able to self-regulate learning. The decision to improve the engagement approach was taken after evaluating the feedback from Cohort 1 that informed, they don"t feel the course instructor engaged with them enough.
The focus group study with both Cohort 1 and 2 provided the detail the practitioners need to effectively know how to improve the design of MS3 module site better. Cohort one"s feedback was generally negative, and they were not satisfied with the 'usability' or how the 'learning' was conducted by the course instructor as well as with the 'technical issues' in TIMeS. The focus group from Cohort 2, on the other hand, provided more positive responses on their experience of 'usability' and 'learning', after improvement was made in the learning design of MS3 to add on opportunities for the course instructor to engage with the learners using Telegram and reply to student"s feedback in Forum and Journal activities. The difference which is the most prevalent between Cohort 1 and Cohort 2 is "teacher presence". Although technical problems still occurred in Cohort 2, it was forgiven due to having the course instructor in close contact with the learners and keeping up with the time window to reply to messages from students within six hours.
It is recommended that a long term study can be conducted on learner"s experience of MS3 or the educational impact as the practitioners continue to improve on the design of the MS3 each semester. Subsequently, a follow-up study can be conducted with other modules sites as well to compare learner"s feedback and how the practice of a systematic way of planning learning design using the LDS Framework can improve learner"s experience in learning entire online for L@S.
The contextualized sampling methods would not make generalization feasible. However, the experience of the practitioners in developing a module site in a Moodle learning environment for L@S can become a guide for practitioners from other learning institutions that also uses Moodle as the LMS. The LDS Framework model used as a learning design tool on the other hand can also be utilized as best practice for designing for L@S when using outcome-based pedagogical approach. Finally, the researcher noted, as a general observation, that the participants of the focus group in Cohort 1 were articulate in expressing their views. However, there were one or two individuals who were more vocal in expressing their views as compared to the rest. Participants in Cohort 2, on the other hand, were a quieter and more reserved group. This was a limitation that was shared in the focus group from other studies [39], [44].

VII. CONCLUSION
In conclusion, the focus group interview was conducted to provide an opportunity to obtain more meaningful data for pre-and post-test implementation assessments (Patton, 1990;Liamputtong, 2015;Penuel et al., 2011). In design-based implementation research (DBIR), implementation is also a primary subject of theoretical growth and study (Fishman & Penuel, 2018). The thematic analysis has enabled the researcher and practitioners to plan out to improve the areas concerned with improving the MS3 design iterations using the LDS Framework model as a tool for learning design. To ensure a positive online learning experience, course instructors must ensure engagement with learners is purposefully designed into the course.

CONFLICT OF INTEREST
The authors declare no conflict of interest. Enna Ayub is the head of Department of the eLearning Academy, a division under the Centre for Future Learning in Taylor"s University, a private university in Malaysia. Before working at Taylor"s University, she has had experience working in the corporate, MNC as well as education sector. She is currently pursuing her PhD in the field of Educational Technology in the Universiti Teknologi MARA, Malaysia (UiTM). She has a master in instructional design and technology and her research interest is on learning design, learning at scale, emerging technologies in education and learning in online environment. Some of her professional affiliations include Association for Instructional Design Advancement (AIDA) and IEEE. She was awarded the President"s Award twice in 2013 and 2016 for her efforts in transformational teaching and learning initiatives in Taylor"s University.
Syamsul Nor Azlan Mohamad is an associate professor in the Faculty of Education, Universiti Teknologi MARA, Malaysia (UiTM). A senior lecturer and a Technologist, he has a Phd in education (IT and computing education) and master"s in educational technology. His area of expertise in curriculum, e-portfolio/alternative assessment, learning and predictive analysis. Now, he is the head of Assessment and Evaluation in Universiti Teknologi MARA. He actively engaged with KPM, KPT, Apple.Inc, MTDC, Celcom-Axiata, MDEC, IBM, HP, Dreamcatcher, public and private universities for research and consultation for digital maker talent and innovation initiative. As for now, he is actively engaged with KPM as an expert and speaker in promoting innovative and alternative assessment in higher education.

Sharon Goh Wei
Wei is the programme director of the dual awards program in the School of Computing and IT, Faculty of Innovation and Technology in Taylor"s University. Her expertise is in e-learning technologies. Her research interests include e-learning technologies, emerging web technologies, critical thinking skills, learning theories, MOOC, ontological based webpage segmentation, tools and applications, social media marketing and communications as well as persuasive messages and communications in social media. Some of her professional affiliations include the International Association of Computer Science and Information Technology (IACSIT) and IEEE.
Johan@Eddy Luaran is an associate professor in education at the Faculty of Education, Universiti Teknologi MARA (UiTM), Malaysia. With more than 15 years of experience in teaching and learning, he has worked in diverse contexts to research and support meaningful learning for students. He obtained his bachelor"s degree in chemical engineering and has a master"s and a PhD degree in educational technology, with a focus on the technological affordances in promoting effective learning. He has written numerous Scopus-indexed papers, led several national projects and has been recognized in various teaching and learning innovations related to e-learning. For these efforts, he has been awarded the UiTM e-Learning Award by the Vice-Chancellor in 2014. He is currently the Director of Advance Learning Division, Institute of Continuing Education and Professional Studies, UiTM who oversees e-content development of the university.