A Checklist Development for Meaningful Learning in Classroom Observation

Classroom observation is generally regarded as an important tool for improving the professional development of teachers. It needs a good checklist for teachers and observers to communicate those performed and missed points to improve teaching practice. However, most of the publicly used checklists tend to lack meaningful learning from classroom observation. To overcome this shortcoming, this study incorporated fundamental professional development pedagogies (e.g., pedagogical knowledge (PK), pedagogical content knowledge (PCK), technological pedagogical content knowledge (TPCK)) to form the checklist, and subjected it to a reliability test (Cronbach's α from .861~.869) and expert validity test to explore its applicability. Moreover, analyzing classroom observation in teaching practices, the results of this study indicated that PCK for lesson planning, PK for teaching method and strategies including cognitive strategies and affective strategies (i.e., in lecturing, in discussion, and in project working) were more applicable to be observed. However, such TPCK, in this study, was discovered that most of the participating teachers had doubts about the applicability of the dimension in the current class observation activities.


I. INTRODUCTION
Learning in the 21st century is to acquire skills for the 21st century, thus involving students' effective use of Information and Communications Technology (ICT) to participate in cooperative tasks and solve practical problems [1]. In this regard, science and technology provide unlimited possibilities for construction, organization, research, visualization, communication and cooperation, evaluation, and all teaching activities [2]. Therefore, teachers must implement various teaching methods and apply ICT as a teaching mode to help students cultivate the required 21st century abilities. The framework of technological pedagogical content knowledge (TPCK) provides the theoretical basis for teachers to use ICT in education [3]. TPCK is based on the long-standing pedagogical content knowledge (PCK) model [2]. Specifically, TPCK divides teacher knowledge into three main parts [4] under the interaction of pedagogical knowledge (PK) and content knowledge (CK). However, TPCK is a dynamic structure, so researchers use different tools and methods to understand the Manuscript received April 9, 2020; revised June 21, 2020. Jon-Chao Hong, Jian-Hong Ye (Corresponding author), and Yu-Ying Yu are with the Department of Industrial Education and Institute for Research Excellence in Learning Sciences, National Taiwan Normal University, Taiwan (e-mail: tcdahong@gmail.com, kimpo30107@yahoo.com.tw, hsu4484@gmail.com).
Po-Hsi Chen is with the Institute for Research Excellence in Learning Sciences, National Taiwan Normal University, Taiwan (e-mail: lorenz1020@gmail.com).
Although science and technology have made rapid progress, teaching and learning in the educational field has not changed at the same pace. There are still many teachers who do not have the confidence and ability to combine science and technology with various teaching tools under different circumstances [6]. Therefore, whether the concept of TPCK theory exists in teaching practice is still controversial [7], and the knowledge needed for teaching is essentially dynamic, not static. In order to understand whether this new knowledge is applied to teaching, researchers need to verify it in new and different ways [8].
For decades, classroom observation has been widely regarded as a tool to improve teachers' teaching efficiency [9]. It may be used for diagnosis, evaluation and consultation, and is regarded as a traditional tool for teaching development processes [10]. Thus, an increasing number of studies have suggested that it is very important to measure the quality of teachers' classroom practice through classroom observation for students' learning effectiveness and other key results (such as cultivating students' social emotional ability) [11]. However, at present, many inspection forms only describe teaching practice, but do not evaluate whether they are consistent with specific teaching strategies [12]. Therefore, designing a classroom observation checklist that can verify teaching strategies would be helpful to improve the quality of teaching evaluation. The first step in designing an observation checklist is to determine the theoretical or conceptual basis, which can be used as the basis for understanding, describing and evaluating teachers' practical results [13].
In the past few decades, people have paid increasing attention to progress, so they encourage the development of standardized observation checklists to better understand and consistently evaluate the teaching practice process [14]- [16]. Related research has called for the development of standardized checklists to determine which course segment has the strongest relationship with students' academic performance in classroom teaching processes such as management, forms and interactions [17].
At present, the common classroom observation checklists in Taiwan emphasize the observation and examination of teaching behavior. In the content of classroom observation, the theoretical basis and theoretical concepts mentioned are relatively weak, and many checklists do not mention the concept of ICT application in the educational field. Therefore, the development of a classroom observation tool based on the TPCK framework will help to understand the current application of educational science and technology and related theoretical theories in the educational field. In this regard, this study carried out the development of the classroom observation checklist in this direction.

II. THEORETICAL BACKGROUND
High-quality teachers should know how to combine subject content knowledge with educational professional knowledge, and with teaching strategies of curriculum design, class management and learning evaluation of various subjects [18]. To design rich scientific and technological learning experience for students, teachers need to have effective skills and knowledge in a scientific and technological teaching environment [19]. Shulman (1986) proposed the viewpoint of pedagogical knowledge growth, and argued that pedagogical knowledge (PK) and content knowledge (CK) interact to produce pedagogical content knowledge (PCK) [20]. In contrast, technological pedagogical content knowledge (TPCK) is teachers' integrated knowledge of content, teaching methods and science and technology in specific field courses [21]. With the development of science and technology, technological knowledge (TK) has been paid increasing attention. Many educational researchers have realized the potential of the TPCK framework. Therefore, TPCK has been regarded as the theoretical basis for developing teachers' constructive use of science and technology to support students' learning and understanding [22]. TPCK is thus regarded as an important theoretical framework in this study. According to the pillar model of teachers' ability [23], the three teaching pillars are described as follows: PK: Pedagogical knowledge refers to knowledge about teaching practices, principles and strategies, and methods needed to manage classrooms [24].
PCK: Teaching content knowledge refers to the knowledge that can choose the appropriate teaching method for the topic. It represents the combination of content and teaching knowledge. It is the knowledge that combines teaching methods and domain knowledge into the curriculum design [24].
TPCK: It is the relevant knowledge that integrates science and technology into teaching methods and content, so that teachers can develop or introduce appropriate teaching strategies according to specific situations and improve students' effective learning [25].
From the perspective of constructivism, knowledge is constructed by students, not directly granted by teachers. Therefore, the effective education method should be that learners learn actively, or have good interaction with teachers or peers to generate knowledge, instead of absorbing knowledge passively or learning alone [26]. In the process of strategy-based teaching, both teachers and students should be able to construct new knowledge [27]. In addition, cognitive strategies support the information processing process in the interaction between learners and learning materials [28], and the strict teaching content is to let students carry out cognitive-oriented learning tasks and activities that are highly related to life [29]. The verification of the cognitive teaching strategy (PK) should thus be emphasized in classroom observation activities.
It was pointed out in a previous classroom observation study that attention should be paid to the classroom environment and teaching [30]. This is because, when teachers and students interact and participate in learning activities, meaningful learning processes will take place, and participatory learning processes will increase information exchange, stimulate interest in learning, and recognize mutual respect between teachers and students [31]. Therefore, if teaching lacks situational factors, then the educational environment will have serious defects [32]. The focus is on teaching methods, teaching skills, use of teaching aids, evaluation of students' learning steps, observation of students' behaviors and expressions, and teacher-student interaction [33]. It can be seen from the above that the situational strategy is also an important influencing factor in teaching activities, and the implementation of this part must also be discussed in classroom observation activities. Therefore, this study regarded situational strategy as one of the important observed constructs.
Teacher competence (Medley, 1982), also known as teaching professionalism, qualifies teachers for their work [34]. This checklist is designed based on a pillar model of teacher competence [35]. Based on the needs of teachers' teaching tasks (PK, PCK, TPCK), the checklist is used to check whether teachers can design courses and teaching plans, formulate teaching strategies (i.e., cognitive strategies, situational strategies), and implement teaching so as to enable students to discuss in groups, work in groups, and apply information technology to classroom practice. After discussing relevant documents, eight constructs of classroom observation were analyzed.

A. Research Process
In this study, K-12 in-service teachers who had participated in classroom observation activities in Taipei City and New Taipei City were invited to fill in the checklist by purposive sampling. A total of 330 questionnaires were distributed to the teachers and collected from May 1 to June 30, 2019.

B. Research Subjects
A total of 330 questionnaires were distributed, and 286 were collected, giving a return rate of 86.6%. After removing 65 invalid data, the valid sample number was 221, and the effective rate of the data was 77.3%. The respondents comprised 71 male teachers (32.2%), and 150 female teachers (67.8%); 15 with an associate degree (6.8%), 87 with a bachelor degree (39.4%) and 113 with a master degree (51.1%), 6 with a PhD degree (2.7%); 93 primary school teachers (42.1%), 69 junior high school teachers (31.2%), and 59 senior high school teachers (26.7%), and they had an average teaching length of 12.64 years (standard deviation of 8.469 years).

1) Preparation of the checklist
Based on the three-pillar model of teachers' competence, the classroom observation checklist developed in this study was developed from previous studies and related theories. The checklist was revised by three professors from education-related institutes and was reviewed by one junior International Journal of Information and Education Technology, Vol. 10, No. 10, October 2020 high school principal and two senior high school principals in terms of content validation. There were found rounds of expert reviews, explained as follows.
The first round of review focused on the design of the appropriateness and completeness of the constructs and their items, and put forward suggestions for revision. The second round of review verified the revision results of the first round, as well as the rationality of the dimensional attribution of the topic items and the readability of the text narration of the topic items, and put forward revision suggestions. The third round of review examined the readability of the revised topic and its annotations, and put forward suggestions for revision. The fourth round of review put forward suggestions for revision of the text fluency and content typesetting of the revised topic. Finally, five teachers who had participated in the classroom observation activities were invited to read the items of the checklist on a trial basis. In addition to personal background data, the contents of the checklist also include: (1) use experience (experienced, inexperienced and uncertain), and (2) applicability (applicable, inapplicable and uncertain) for selection and filling. The relevant constructs and items are described as follows.

2) Pedagogical content knowledge (PCK)
The PCK construct emphasizes the activity design, learning theory and design activities used by teachers according to the characteristics of learners for other teachers to observe. Relevant items are shown in Table I.   TABLE I: TEACHING PLAN DESIGN ITEM CONTENT  Coding  Theme content  PCK-1 This teaching drew on learning theories (such as situated-oriented and inquiry-oriented) to design the teaching activities. PCK-2 The design of the teaching consequences is logical. PCK-3 The teaching is planned according to students' cognitive style and ability. PCK -4 The teaching is designed according to students' learning types (visual, auditory, tactile, etc.). PCK -5 The teaching design is school-based. PCK -6 The teaching design is outcome-based (learning objectives: memory and understanding). PCK -7 The teaching design is inquiry-based. PCK -8 The teaching design is to integrate cross-domain knowledge. PCK -9 The teaching design is project-based (e.g., How to build a house). PCK-10 The teaching design is problem-based (e.g., How to solve the water leakage in the house).

PCK -11
The teaching design is based on students' diversified intelligence levels.

3) Technological pedagogical content knowledge (TPCK)
The TPCK construct emphasizes the situation in which teachers apply educational technology or multimedia in the teaching field in the activities of classroom observation. The relevant items are shown in Table II.   TABLE II: TPCK CONSTRUCT ITEM CONTENT  Coding Topic content TPCK-1 Use appropriate digital media materials to provide students with understanding of the teaching content. TCPK-2 Self-made digital media materials provide students with understanding of the teaching content. TCPK-3 Combine online social media (e.g., YouTube) to provide students with effective learning.

Coding
Topic content TPCK-4 Combine search websites (e.g., Google) to provide students with effective learning. TPCK-5 Use new information technology (e.g., VR, AR) to provide students with effective learning. TPCK-6 Use the network blended teaching mode (e.g., MOOCs) to provide students with effective learning. TPCK-7 Use the multi-media to carry out different flipped teaching (e.g., Front flip and back flip teaching methods) to provide students with effective learning. TPCK-8 Use digital game devices to provide students with effective learning. TPCK-9 Use technology to identify students' learning disabilities and provide scaffolding to improve their learning performance.

4) Cognitive strategies in learning (CSL)
Pedagogical knowledge related to cognitive strategies in lecturing (CSL) emphasizes that the teacher takes into account the learners' cognitive ability or cognitive activities in the classroom observation activities. The relevant items are shown in Table III. During the lecture, the teaching method can be adjusted appropriately to guide students to understand deeply. CSL-6 Can detect students' cognitive fatigue in class and change teaching methods. CSL-7 Provide cognitive scaffold (from simple prompt to in-depth explanation) for students to effectively learn. CSL-8 Pay attention to students' cognitive load and adjust the difficulty of the teaching materials. CSL-9 Combine different evaluation methods to assess whether students understand the content of the class. CSL-10 The teaching is aimed at students' misconceptions and changes teaching activities. CSL-11 The teaching has the effect of strengthening working memory. CSL-12 The teaching used specific sensory images to increase the effect of cognition.

5) Cognitive strategies for project working (CSP)
Pedagogical knowledge related to cognitive strategies for group project-working (CSP) focuses on the situation in which teachers can guide learners to carry out cognitive learning activities when explaining topics in classroom observation activities. Relevant items are shown in Table IV.   TABLE IV: CSP CONSTRUCT ITEM CONTENT  Coding  Topic content  CSP-1 Guide students to analyze the similarities, differences and correlation of information and establish their knowledge structure. CSP-2 Guide students to visualize concepts and organize knowledge into charts. CSP-3 Based on laws of practice in teaching materials, students can master knowledge of the "anchoring concept" through repeated exercises, but they will not study too much. CSP-4 Guide students' memory skills and make students remember the learning content efficiently. CSP-5 Guide students to self-perceive mistakes and to think about ways to improve (cultivate metacognitive ability).

6) Cognitive strategies for discussion (CSD)
Pedagogical knowledge related to cognitive strategies for group discussion (CSD) focuses on checking the cognitive activities of teachers in teacher-student discussions or peer discussions of learners during classroom observation activities. Relevant items are shown in Table V.   TABLE V: CSD CONSTRUCT ITEM CONTENT  Coding  Topic content  CSD-1 Guide students to speak with evidence about what their predecessor said. CSD-2 Give students the opportunity to think and express themselves in multiple ways. CSD-3 Detect students' stereotypes and change their mental model appropriately.

CSD-4
Guide students to have constructive (positive) critical dialogue. CSD-5 Guide students to express their ideas in an organized and hierarchical way. CSD-6 Guide students to keep their speaking focused on the current topic. CSD-7 Guide students not to say the same thing again and again, but to have new ideas when speaking again.

7) Affective strategies for learners (ASL)
Pedagogical knowledge related to affective strategies in learning (ASL) focuses on checking the situational teaching methods adopted by teachers in classroom observation activities, and on the situation where learners' feelings are expressed in changing teaching methods. Relevant items are shown in Table VI.   TABLE VI: ASL CONSTRUCT ITEM CONTENT  Coding Topic content ASL-1 Identify students who are anxious in their studies, adjust teaching methods in time and relax the class climate. ASL-2 Able to adjust teaching methods in time to maintain students' interest in the lesson. ASL-3 Notice absent-minded students and adjust teaching methods. ASL-4 Notice students who do not understand but pretend to understand and give opportunities to enhance their self-confidence (e.g., Non-verbal support).

8) Affective strategies in discussion (ASD)
Pedagogical knowledge related to affective strategies for group discussion (ASD) emphasizes that teachers will adopt corresponding teaching strategies according to the current situation or the perceived affective performance of students in teacher-student discussions or peer discussions of learners in the course of classroom observation activities. Relevant items are shown in Table VII.   TABLE VII: ASD ITEM CONTENT  Coding Item content ASD-1 Guide students to effectively (not chatting off-topic during the discussion) have classroom discussions. ASD-2 Guide students not to talk for too long or just for the sake of talking. ASD-3 Guide students who evade speaking to speak confidently. ASD-4 Guide students who speak to focus on the current topic. ASD-5 Guide students not to use too many negative emotional words when speaking.

9) Affective strategies in project working (ASP)
Pedagogical knowledge related to affective strategies for group project-working (ASP) emphasizes that teachers can induce and inspire learners' positive affective reactions and promote learners to increase their learning initiative when explaining topics in the activities of classroom observation classes. Relevant items are shown in Table VIII.   TABLE VIII: ASP ITEM CONTENTS  Coding Item contents ASP-1 Guide students to self-regulate their learning and to do effective previewing before class. ASP-2 Guide students with negative mentality (e.g., Blaming themselves or others when they make mistakes) to have positive thinking. ASP-3 Notice free-riders in teamwork (those who do not really participate in learning); give them support and guide their participation. ASP-4 Teaching activities are designed with goal achievement of motivation to stimulate students' active learning attitude.

A. Reliability Analysis of Each Construct
Cronbach's α is one of the most widely used reliability test indicators in social science, and is used to prove that the scale adopted conforms to the research purpose. Scholars generally agree that the Cronbach's α value must be greater than .70 or higher, which can be regarded as reaching the acceptable standard (Taber, 2018) [36]. In this study, the values of each construct are between .761 and .869, while the value of the whole checklist is .943. The eight constructs developed in this study therefore have good reliability.

A. Meaningful Learning for Teaching Professional Development
Classroom observations can be used for diagnosis, assessment, and counseling, and can be considered as traditional tools planned for teaching development [10]. The first step in designing an observation inspection system is to determine theoretical or conceptual foundations that will provide a basis for understanding, describing, and evaluating teachers' practices [13]. Therefore, based on the TPCK, this study explored relevant literature, and analyzed that in the course of teaching practice, teachers should pay attention to categories such as teaching plan design, science and technology teaching content knowledge, cognitive strategies and situational strategies. In the process of developing the checklist, the eight classroom observation constructs were further subdivided.
The classroom observation activity is to confirm that ICT has been integrated into the teaching [39]. This study found that participants' views on the applicability of TPCK in classroom observation activities are relatively different. It was uncertain whether some of the questions apply. This is similar to the research results of Chai and Koh (2017) using the TPCK framework. Chai and Koh pointed out that, to a large extent, teachers still focus on replacing teacher-centered teaching in ICT integration, rather than changing teaching methods to support learning in the 21st century [40]. Moreover, many researchers have pointed out that it is difficult for teachers to integrate science and technology into their teaching process [41], resulting in a large number of research reports showing that the expected teaching changes from ICT to teacher-led learning to student-centered learning are still not common in schools [42], [43].

B. Meaningful Learning for Development of Academic Specialty
Teachers' continuous participation in professional learning activities is very important to improve their knowledge, teaching and students' learning (Akiba & Liang, 2016) [44]. At present, in large-scale research, there is a lack of consistent findings to study the relationship between teachers' knowledge classroom practice [45]. Therefore, in the curriculum evaluation system, an increasing number of observational measures (e.g., checklists) are used to ensure that the observed curriculum activities can fully reflect the changes in teachers' practice [46].
As the 12-year Basic Education curriculum has been introduced, efforts should be made to emphasize the importance of core literacy such as autonomous learning, communication and interaction, and social participation. Among them, scientific and technological information and media literacy in the construct of communication and interaction advocate that learners should have the ability to make good use of science and technology and information media [47]. This study is based on the curriculum checklist developed according to the TPCK framework, and TPCK's teaching theory coincides with the teaching concept proposed in the 12-year Basic Education Curriculum Outline that has just been implemented. Educators suggest that only when teachers have teaching literacy can they effectively develop students' core literacy. Therefore, the meaning of "literacy" should be used to plan, formulate and implement literacy-oriented educational objectives, courses, teaching, learning evaluation and academic guidance [48].
Moreover, it is also pointed out that in terms of curriculum design, in addition to the professional standards of teachers, teachers must also conform to the 12-year Basic Education's new curriculum concept, quality-oriented teaching characteristics, and the core competence of teachers, curriculum learning content, curriculum learning activities and curriculum assessment so as to bring students a core literacy-oriented learning process [18]. Therefore, the checklist developed in this study is helpful for researching teacher training, or the practical level of teacher professional development activities of serving teachers.

VI. CONTRIBUTIONS AND FUTURE RESEARCH
With the development of teaching methods and theories, as well as changing pressures, the types of classroom observations applied in research should evolve over time [49]. Under the TPCK framework that emphasizes the application of ICT, this study developed a checklist to observe the design of teaching plans, knowledge of science and technology teaching content, cognitive strategies and situational strategies, which may help to understand the discussion and application of technology, learning theory and related strategies in a bid to provide participants with meaningful classroom observation activities.
As this study focused on the development of the classroom observation checklist, it emphasized the appropriateness of the content of the eight-construct observation activities. Therefore, no further empirical research was performed using the scale assessment. The Likert scale is a common scoring format in surveys. Respondents were ranked in 5 or 7 levels from high quality to low quality [50]. In the follow-up research, a 5-point scale design method can be used to allow teachers participating in classroom observation activities to fill in the "easiness of observation" and "needs of core literacy-based classroom activities" of the teaching activities explained through each item so as to understand the actual situation in the current teaching scene.