A Learning Design Strategy Framework for Content Transformation Using Fuzzy Delphi Method

This study engaged experts in confirming the elements and items of a Learning Design Strategy Framework (Framework) for content transformation of learning at scale (L@S) to a virtual classroom; aligned with the new Taylor’s Curriculum Framework (TCF). In ensuring the practitioners can work collaboratively for rapid content transformation; a Framework is deemed necessary for a systematic campus-wide content transformation effort. This study aims to report a systematic approach in prioritizing elements and items of the Framework’s matrix for the practitioners to utilize when transforming their modules as fully online or highly blended mode for L@S. Using the iterative development of the Malaysian Studies 3 (MS3) module site as the exemplar module to test the Framework, changes were made to the elements and items in the learning design matrix, according to the priority of the TCF, using the Fuzzy Delphi method. In Delphi round 1 and Delphi round 2, the experts’ feedback was sought. The findings of this study assisted to finalize the elements and items and its sequence for the Framework to ensure alignment with the TCF at module level. This Framework can be used to guide future content transformation efforts when change management must happen rapidly to cope with disruptive education. The mismatch in prioritized items between expert consensus, the researcher and in-house experts revealed factors that influenced the prioritization process and practicality to operationalize rapid online module site development. More research should be conducted after the MS3 was implemented to investigate learner’s reaction having to learn fully online and how that affects their satisfaction and the relevance of the mode of learning to their lifestyle as a 21st Century Learner.


I. INTRODUCTION
To remain competitive as a private higher education institution, Taylor"s University (TU) stride a bold move in ensuring the education offered to its students is synchronous to the change that happened globally due to disruptive education brought forth by the Industrial Revolution 4.0 (IR4.0). In March of 2018, Taylor"s University launched its new Taylor"s Curriculum Framework (TCF) that allows its learners to be empowered to design their own education path, based on their chosen specialization, and combined with various interest in other fields. The transformative pedagogy Manuscript received April 7, 2020; revised October 9, 2020. This work was supported in part by the Taylor"s University, TRGS/ERFS grant.
E. Ayub is with Universiti Teknologi MARA, Kuala Lumpur Malaysia. She is also with the eLearning Academy in CFL, at Taylor"s University, Malaysia (e-mail: enna.ayub@taylors.edu.my).
S. N. A. Mohamad  under the TCF further ensues personalized and self-directed learning. Further, with the capability of disruptive technology, learning can now be delivered at scale in the online learning environment, no longer confined to a traditional brick and mortar classroom. While online learning is no longer a nascent field, the delivery of learning at scale (L@S) is still a new body of knowledge in Malaysia"s higher education landscape and needs to be investigated. Additionally, the change that happens in TU because of TCF, rippled at multi-levels across campus to ensure other faculties, schools, departments, and units are also aligned to the TCF. The TCF phase one affected change at policy and program level, while the TCF phase two affected change at module level. As part of the effort to support the practitioners among the e-Content Development Specialists and the educators, working collaboratively to transform the modules to the online learning environment rapidly in TCF phase two for year one implementation by 2020, having a Learning Design Strategy Framework (Framework) for online content transformation is deemed necessary to guide the rapid transformative process that is also constructively aligned with the TCF. This study was conducted in iterations, in a period of two years from 2017 to 2019, to come up with a Framework to assist the transformational process based on the development of the Malaysian Studies 3 (MS3) module site in TIMeS as the exemplar.

II. BACKGROUND
A. The Education 4.0 Learning Environment for L@S Having the right disruptive education tools assisted in the delivery of learning in the virtual classroom. More importantly having a pedagogical strategy will ensure the tools are applied strategically to ensure learning in an effective manner can happen for L@S. Previous research on L@S discussed MOOC of its bad design [1], [2]. Further, the research conducted by the researcher on teacher presence, based on the Technopreneurship MOOC taken by the students of Computing and IT from TU, revealed learners expect their course instructors to be highly engaging with them in the online learning environment just as they have been in the face-to-face setting [3]. Because of the case-study conducted in TU, gave reason to doubt MOOC pedagogy, the University under the directive of the eLearning Academy (e-LA) are currently more careful in using MOOC due to the criticism received from academics and learners as informed by studies example from [3]- [5]. However, it is worth to note, conducting a course in virtual classroom is still an approach to be considered for L@S because of its convenience. Hence, this study focuses on delivering L@S just for the Taylor"s University"s learners. This method of learning online with a massive number of learners using TIMeS has never been implemented in TU. When designing for a new adoption of technology, the new implementation must be "simple, intuitive and pleasurable", for example, putting usability as the forefront instead of aesthetic appeal (p. 4) [6]. Hence, to effectively design for L@S, it is imperative for the practitioners to plan the learning design strategy of the module with the learners in mind particularly on how engagement shall happen between the instructor, the learner and their peers.

B. The Need of a Learning Design Strategy Framework
The launch of the new Taylor"s Curriculum Framework (TCF) in Taylor"s University in March 2018, brought upon ripple effect at multi-levels across campus [7]. Reference [8] informed the process of constructive alignment of the TCF is challenging due to the iterative process of alignment at program and module levels (p. 111). Further, upon investigation, based on the design-case of the development of the MS3 module site exemplar, the course instructors are not ready to develop the MS3 module site on their own and need guidance from the e-Content Development Specialists [9]. Furthermore, findings from the analysis of the e-Content Development Specialists informed they do not have a system for learning design to ensure their learning design strategies are aligned with the TCF. Since the TCF is also new, many practitioners do not know the details of the TCF. A Learning Design Strategy Framework (Framework) will assist the practitioners among the educators and e-Content Development Specialists to cope with the rapid changes under the TCF and to ensure the learning delivered fully online for L@S, through the TIMeS module sites is constructively aligned with the TCF [10]. Further, having a Framework ensures the practitioners can transform their module sites through careful planning and reflection, putting thoughts into a design proof in the learning design matrix of the Framework. Further, having a clearly defined Framework ensures pedagogy ideas can be shared, discussed upon and refined when developing the module site collaboratively among the practitioners [11].
Reference [12] theorized that the design of a well-developed framework must be in cognizant with learning theories and pedagogical approaches developed rigorously through well-informed research. However, because the Framework was not readily available at the start of the transformative change of the newly implemented TCF, and due to the urgency to operationalize the learning and teaching of the MS3 module in the online learning environment, the Framework had to be developed in iterations with expert"s recommendation and careful reflection of the researcher with the assistance of the practitioners in this project, throughout the development process of the Framework in synchronous with the development of the MS3 module site in TIMeS. The learning design strategies presented in this study unites previous research on learning design framework for L@S yet has its distinctive characteristics to constructively align to the pedagogy under the TCF.

III. THE SYSTEMATIC LITERATURE REVIEW (SLR) STUDY
A. Identifying the Pedagogical Strategies in Designing for L@S An investigation of pedagogical strategies includes learning theories, approaches, modes, methods, perspectives, models, characteristics, and application was conducted prior to the Fuzzy Delphi Method (FDM) to anchor the learning design strategy elements and items for the Framework to be developed. The approach to conduct a systematic literature review (SLR) using the PRISMA method was implemented to form the underpinning pedagogical strategies based on the principles of an effective module design for L@S [13]. An explorative mind-set was applied by the researcher to re-examine the scholarly studies published on Frameworks used for collaborative learning design process to form the blueprint or design proof of a module to be conducted for L@S. In this study, this Framework will be utilized as the tool used for collaborative learning design strategies between the practitioners in the design team (e-Content Development Specialists) and those from the educator team (course instructors).
In identifying the pedagogical strategies for the Framework, such as the pedagogy theories, models, perspectives, approaches, modes and methods in designing for L@S, the PRISMA method was used by performing a keyword search applied in electronic database, thesis, journals, conference proceeding, and a scan through was made for relevant articles in the author"s reference lists. Next, the following key words were winnowed through by researching literary works published between 2013-2018 "learning design approach" (es), "pedagogy models for e-Learning", "pedagogy approach" (es), "Education 4.0 pedagogy modes", "Learning design strategy framework", "Learning design model", "Learning massive online course", "learning design MOOC", "learning design VLE". Due to the limited number of research on the learning design for learning at scale (L@S), a more general term such as "Learning Design" or when mentioning L@S, the term MOOC was used as a keyword for the search such as "Learning Design MOOC". Further, due to the lack of recent literary works on the items searched, the year of the search was extended from 2010 to 2018. In addition, since the search is focused on "learning design strategy framework", the following keywords were excluded "instructional design approaches", "instructional design strategies", model or framework. This was due to the scholarly argument that a distinction must be made between "learning design" and "instructional design" [13]- [15].

B. Synthesizing Theory and Research
The search results for the keywords mentioned returned 114 papers. Twenty-five papers were selected based on the search aimed at finding scholarly works that were most closely related to pedagogical models and learning design. Based on Gráinne Conole"s approach to associating relationship to pedagogy or learning theory, the objective is to expand the results of her literature review of pedagogical strategies of learning design to the context of the development for L@S that is constructively aligned to the TCF [16]. After careful consideration, this comprise of perspective, approach, characteristics, and type of approach which can be inclusive of models, frameworks, and application in the context of online delivery in the existing review. While the initial SLR evidenced the proposed principles for the elements and items of the Framework based on academic research, an FDM study will follow suit to obtain expert"s consensus for the Framework.

A. Fuzzy Delphi Method
The Fuzzy Delphi Method (FDM) was adapted to verify the elements and items and its sequence in a matrix for the Framework due to time constraints to ensure the modules transformation to the fully online mode supports TCF phase 2 for March 2020 execution campus-wide. MS3 is developed as the module site exemplar in TIMeS. The need for rapid development of the Framework is due to rapid technological advances today may invalidate development work before it is implemented [17].
The reliability and validity method of the FDM is by using a particular type of fuzzy sets called Triangular Fuzzy Numbers. While the classical logic of number is crisp and certain, in fuzzy logic, each number has an approximate value [18]. According to reference [18], the advantage of using the traditional Delphi Method (DM), while is useful to obtain expert"s feedback, can be time consuming, whereas the development of the Framework must be developed rapidly to cope with the TCF implementation at phase 2. Hence, a modified process to the Fuzzy Delphi Method (FDM), was applied to assist rapid development of the Framework yet able to reduce ambiguity, diversity and discrepancies of opinions among the experts which requires repetitive rounds, often reported of the traditional DM [19]. This same FDM was also applied in research conducted to develop e-portfolio elements in art and design context [20], [21] (See Fig. 1 of the FDM process applied for this study). Some similarities to the traditional DM was maintained in FDM, such as the procedure of maintaining the anonymity of the experts to elude any form of group thinking, using questionnaires [19]. The experts" consensus was sought using a set of algorithms in the FDM process, where the applicable sets of learning design strategy for a Framework derived from the SLR, will be used to develop the Malaysian Studies 3 (MS3) used as the exemplar module site for L@S.
In determining the size of the panel of experts, the opinions of [16] and [22] were agreeable to be applied for this study. Their studies recommended between five to 10 experts from different learning institutions with specialized knowledge. In this study, the experts were selected from among those with specialized knowledge and different specialties in Learning Design, Educational Technology, e-Learning, MOOC and L@S, so that they can provide different yet constructive perspectives for the Framework in a rapid yet efficient manner to achieve a cohesive conclusion [23].

A. Phase 1
The data collected from the Systematic Literature Review (SLR) process was analysed using thematic analysis, six steps approach by [24] for the learning design strategies (See Fig. 2). The researcher as the human instrument reflected on the themes aligned with the TCF and method generated by [24] and listed the learning design strategies relevant for the Framework such as the learning perspectives, the author(s) and date of publication, approach, the model or framework, the type of approach, characteristics and application in online learning (as tabulated as Table I). Subsequently, a draft of the proposed learning design strategy template and the protocols for a semi-structured interview and sets of questionnaires with open-ended questions for Round 1 FDM was prepared.
For greater rigor, the protocol for the questionnaires for Round 1 FDM, was piloted to test and adjust the questions to eliminate potential procedural problems [25]- [27]. Further this pilot process was added to align with the TCF, hence the in-house experts" opinions were sought. Considering the urgency to develop a Framework applicable with the TCF, purposeful and convenience sampling method were used and minimal n=3 experts participated in the pilot, they are identified as Expert 1, Expert 2 and Expert 3 [27]. It is also important to note here that Expert no 1"s insight gave more weight to all other expert"s due to the context of the study and the Expert is the policy delegate of the TCF [23]. The researcher"s method have to be verified by Expert 1 for alignment. In order to eliminate element of biasness, the researcher"s method, verified by Expert 1 must be verified by all other experts, first in the pilot FDM by in-house expert and then with all other experts. Meanwhile, the in-house Expert 2 and 3 insights from their respective field of specialization, i.e. training of eLearning and pedagogy and pedagogy and instructional design technology as applied in TU"s context for TCF will further assist in refining the protocols. Because the experts are busy people, they were given a choice for an interview or to fill in an online survey tool, survey monkey. Fig. 2. Thematic analysis stages [24].
After the refinement of the protocols and questionnaires with the expert from the pilot Delphi, the interview protocols International Journal of Information and Education Technology, Vol. 10, No. 12, December 2020 were refined and the survey with 23 questionnaires were sent to other experts outside of TU. All experts were given some information on the TCF, the graduate capabilities practiced in TU and Taylor"s pedagogies, the portions which were allowed to be shared by Expert 1. Because of the urgency to have a Framework, and experts are generally busy people, the decision was made for the survey to be sent out using the online survey tool, Survey Monkey. The experts came from two sectors: academia and industry. The insights of the experts were processed using the Atlas.ti V8 tool to obtain the themes for the elements and items for the Framework. N=23 experts responded to the survey, however only n=7 expert"s opinions were considered. As part of the filtration of expert is the number of years working and highly educated (PhD holders) in the field. Further n=13 responses were incomplete and n=3 have less than five years working experience and not considered highly educated in the field.
The results of the thematic analysis from all the 10 experts is as per the constructs in Table III. It is important to make a note at this point to the reader, the process of the refinement of the Framework and the development of the Malaysian Studies 3 (MS3) module happened interdependently. The design proof used to develop MS3 in iteration was a product of the refinement of the Framework.
As a summary, in Round 1, a group of experts gave their opinion on the elements and items for the Framework to develop L@S based on the SLR conducted by the researcher. Open-ended questions collected their insights for the Framework"s best practices. Table II summarizes the collective pick of learning models, perspectives, approaches, sequence of elements as well as insights. The numbers in the table denotes the sequence of the items to appear on the Framework"s matrix. Next, in Round 2, another set of experts will give consensus based on the items presented to them in a set of questionnaires followed by a defuzzification process.

B. Phase 2
In the FDM Round 2, The Learning Design Consensus questionnaire was designed and administered to experts with the same criteria as Round 1. These are different set of experts and their role was to provide consensus on the elements and items suitable for the Framework selected from the experts in Round 1. A 5-point Likert scale anchored between strongly disagree and strongly agree. 20 experts answered the survey with only 19 valid returns. However, only 16 experts" opinions were considered because of their in-depth experience and level of education. Step 1: After receiving the invitation to answer the survey, the experts answered a survey containing a 5-Point Likert Scale questionnaires to decide the importance of the criteria to be included as the elements and items of the Framework using variables as Table III. Step 2: Experts choose the importance of the elements In this step, the experts make decision of the importance of the elements and items based on the weight of criteria. 5 variables to choose from the Likert Scale are "Strongly Disagree", "Disagree", "Not Sure", "Agree, "Strongly Agree".
This research utilized the simplified method of systematic method to calculate FDM adopted from reference [21] using Microsoft Excel. The method embedded a formula developed by reference [19], using three strings of real numbers (F= (l, m, u)), illustrated in Table II as the fuzzy set of Triangular Fuzzy Numbers to assist on the reliability and validity of the data analysis process. This same FDM technique was also applied by reference [20] and [18]. The formula below was embedded in Microsoft Excel:

( )
Let fuzzy numbers be the rating of alternative i with respect to criteria and be the jth criteria weight of the kth expert for i=1… m, j=1,..., n, k=1, ..., K. and K is the number of experts. Let fuzzy numbers be the rating of alternative i with respect to criteria and be the jth criteria weight of the kth expert for i=1… m, j=1... n, k=1, ..., K. and ( ) The distance between two fuzzy numbers are computed as: ̌ ( ) and ̌ = ( ) is computed by, Step 4: Specifies the threshold value (value of d) If upon calculation, the value d is d <0.2, then assume all experts have reached a consensus If upon calculation, the value of d> 0.2, the researchers have to repeat the procedure.
Step 5: Obtain 75% consensus At this juncture, a consensus of the group of experts is achieved if consensus is greater than 75%, while if less than 75%, the researcher can repeat the process.
Next, a defuzzification process is applied to achieve ranking. This ranking is to determine the importance of the elements and items based on expert consensus. The process International Journal of Information and Education Technology, Vol. 10, No. 12, December 2020 885 Step 3: Get an average rating Step 6: Perform a Fuzzy evaluation involved using a formula using complex numbers as alternative to using crisp mathematical numbers. For this study, the researcher used the following formula, embedded in Microsoft Excel: Aggregating the fuzzy evaluations as: Where ̌ X + X + . . . . . . . . .

X And ̌ is defuzzified by
The ranking is determined by the value of Step 7: Researcher"s Reflection and Member Check However, based on the background of this design-case, because of the unique context of the TCF, and also because the TCF policy documents is confidential, the researcher"s reflection along with member check by the in-house experts were applied as a final analysis method to review the findings from the expert"s consensus and to conclude the Framework.
VI. FINDINGS Table III shows a summary of the expert"s consensus percentage, rejected items from the constructs. Items A6, B1, B2, C2, F3, F4, F5, F6, F7, F9, F12, G2 and G5 exceed the threshold value (d) = 0.2. Meanwhile, F5 and G2 were not accepted as the items did not reach the above α-cut=0. 5. Constructs (the elements) B, F and G shows percentage of consensus among experts was less than 75%. For constructs A, C, D and E, the value indicated more than 75% consensus was achieved. Further, the threshold did not exceed 0.2 of agreement among experts. Therefore, the constructs A, C, D and E are accepted as the elements for the Framework by the experts" consensus. All exceeded thus not accepted items according to the FDM technique were reviewed individually by researcher and upon reflection of its necessities to be inclusive in the Framework to align to the TCF, A6 (0.613), B1(0.563), G2 (0.421) and G5 (0.613) were rejected due to its low score in the respective constructs. Finally, member check was conducted with in-house experts n=4 for the relevance of the constructs and items to the TCF, evidenced in Table III. For the member check, Expert 1"s opinion was not sought to avoid bias due to his role as the TCF policy delegate. A slight disparity occurred in prioritized of the items between the expert consensus, the researcher and in-house experts revealed factors that influenced the prioritization process. The prioritization of items for the Framework have more similarities in opinion than the sequencing of the items. Reason being the expert may not fully understand the context of the TCF with having only partial information about the items extracted from the TCF by the researcher, which is the importance of having the module learning outcomes, assessment and the graduate capabilities aligned together as a triage, and the adopted TU pedagogies as some of the TCF guideline. Further the TCF itself in its implementation can be quite complex to comprehend even for the educators. On the other hand, the expert"s opinions if only reflected by the researcher who is the human instrument may also be subjected to potential bias. As an additional measure to ensure rigorousness of the method, the in-house expert from TU was invited for peer-review as a form of member check to verify all items selected through the consensus and reviewed by the researcher are accepted as the final Framework version. The finalized Framework elements and items is as illustrated in Table IV. As a contribution to the body of knowledge, the systematic approach and the FDM process applied for the selection of elements and items of the Framework for L@S to be aligned to the University"s curriculum can be used for validation in future studies when change management has to happen rapidly to cope with disruptive education 4.0 (Edu 4.0) brought forth by the industrial 4.0 revolution. Further, the final version of the Framework can be used as a template for learning design strategy and be shared online, so that practitioners from other higher education institutions intending to leap to Edu 4.0 can apply same process as a guide for their own content transformation process. On the other hand, a follow up research should be conducted to investigate the effectiveness of the application of the Framework as a stepping-stone in the transformation of F2F content to the fully online environment for L@S from the practitioner"s perspective to guide them in planning the learning design strategies for content transformation rapidly yet effectively. Finally, a participative approach towards the development of a module site using the MS3 by including the learners to evaluate their experience of learning in MS3 in the fully online environment as a result of planning done using the Framework can also be explored.

CONFLICT OF INTEREST
The authors declare no conflict of interest.
AUTHOR CONTRIBUTIONS Enna Ayub a PhD candidate and the principal researcher of this study. Any updates of this study such as the final design of the Framework in question is available at: http://www.ddresearch.org/; Associate Professor Ts. Dr. Syamsul Nor Azlan Mohamad is one of the research supervisors and lend the instrument to calculate the FDM he developed using Microsoft Excel; Dr. Goh Wei Wei is one of the research supervisors and co-researcher, often acts as a member check for any interviews conducted for the research; and Associate Professor Dr. Johan@Eddy Luaran is also one of the research supervisors and provided the overall guidance of the research. Copyright © 2020 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0). Enna Ayub is the head of Department of the eLearning Academy, a division under the Centre for Future Learning in Taylor"s University, a private university in Malaysia. Before working at Taylor"s University, she has had experience working in the corporate, MNC as well as education sector. She is currently pursuing her PhD in the field of Educational Technology in the Universiti Teknologi MARA, Malaysia (UiTM). She has a master in instructional design and technology and her research interest is on learning design, learning at scale, emerging technologies in education and learning in online environment. Some of her professional affiliations includes Association for Instructional Design Advancement (AIDA) and IEEE. In 2019, she led her team to pitch this project, using the M3 module site as the exemplar in the Minister of Education Malaysia: Curriculum Design and Innovative Delivery (AKRI) competition and received 1st place under the category immersive learning.
Syamsul Nor Azlan Mohamad is an associate professor in the Faculty of Education, Universiti Teknologi MARA, Malaysia (UiTM). A senior lecturer and a Technologist, he has a Phd in Education (IT and Computing Education) and Master"s in Educational Technology. His area of expertise in curriculum, e-Portfolio/alternative assessment, learning and predictive analysis. Now, he is the head of Assessment and Evaluation in Universiti Teknologi MARA. He actively engaged with KPM, KPT, Apple.Inc, MTDC, Celcom-Axiata, MDEC, IBM, HP, Dreamcatcher, public and private universities for research and consultation for digital maker talent and innovation initiative. As for now, he is actively engaged with KPM as an expert and speaker in promoting innovative and alternative assessment in higher education.

Sharon Goh Wei
Wei is the programme director of the Dual Awards program in the School of Computing and IT, Faculty of Innovation and Technology in Taylor"s University. Her expertise is in e-learning technologies. Her research interests include e-learning technologies, emerging web technologies, critical thinking skills, learning theories, MOOC, ontological based webpage segmentation, tools and applications, social media marketing and communications as well as persuasive messages and communications in social media. Some of her professional affiliations include the International Association of Computer Science and Information Technology (IACSIT) and IEEE.
Johan@Eddy Luaran is an associate professor in education at the Faculty of Education, Universiti Teknologi MARA (UiTM), Malaysia. With more than 15 years of experience in teaching and learning, he has worked in diverse contexts to research and support meaningful learning for students. He obtained his bachelor"s degree in chemical engineering and has a master"s and a PhD degree in educational technology, with a focus on the technological affordances in promoting effective learning. He has written numerous Scopus-indexed papers, led several national projects and has been recognised in various teaching and learning innovations related to e-learning. For these efforts, he has been awarded the UiTM e-Learning Award by the Vice-Chancellor in 2014. He is currently the Director of Advance Learning Division, Institute of Continuing Education and Professional Studies, UiTM who oversees e-content development of the university.