Engineering Investigation of Value Trace Problem for C++ Programming Self-study of Novice

 Abstract —Presently, C++ programming has been taught in many universities around the world as the first object-oriented programming language for undergraduate students to start studying programming concepts and computer architecture. However, many of them may struggle in studying C++ programming due to the nature in the formal language. Thus, hands-on self-study tools can be effective. In previous, we have developed Java programming learning assistant system (JPLAS) for assisting self-study of Java programming. JPLAS offers the value trace problem (VTP) for novice students to study Java programming through code reading study. In a VTP instance, actual values of important variable or standard output messages in a given source code are questioned, where the correctness of each answer is checked through string matching. In this paper, we investigate the effectiveness of VTP for hands-on self-study of C++ programming by novice students. We generated 37 VTP instances for basic grammar concepts using source codes in textbooks or websites for C++ programming, and asked 46 students in three universities in three countries to solve them using the answer interface for JPLAS. The results suggest that most of the students are satisfactory, but some students need cares at early programming study


I. INTRODUCTION
The programming language C++ has been popular even now as a general purpose language that has been designed to making programs in the variety of application domains. According to the IEEE Spectrum [1], C++ is currently the fourth popular programming language, even though long time has passed since the appearance. C++ is an extension of the traditional programming language C, to offer the advantages of the object oriented programming. C++ keeps the functional features of C to facilitate the low level manipulations of computers by programs. As a result, C++ programming is broadly taught as a fundamental programming language in many universities around the world.
From viewpoints of programming educations, C++ can be a good start for studying programming from the ground up if compared with more advanced and practical ones such as Java, Python, and Ruby. C++ programming supports Manuscript  accessing memories and registers in computers by programs for low level manipulations, which is also useful in studying the computer architecture. At the same time, it allows the object oriented programming features as the more advanced and abstract programming. C++ is actually used in almost every area of software developments, such as application software, programming languages, game developments, and embedded systems. However, many students are still struggling in studying C++ due to the nature in the formal language. Hands-on self-study tools can be effective to solve it.
In previous, we have developed Java programming learning assistant system (JPLAS) for assisting self-study of Java programming [2]. JPLAS provides several types of exercise problems to assist Java programming studies at various levels. One of them is the value trace problem (VTP) that is designed for novice students to study basic grammar concepts and programming skills through code reading [3]. A VTP instance consists of a source code, a set of questions, and correct answer strings. Each question requests to answer the actual value of an important variable or a standard output message in the source code. The correctness of any answer from a student is marked through string matching with the stored correct answer string in the web-based offline answer interface [4].
In this paper, we investigate the effectiveness of the value trace problem (VTP) for hands-on self-study of C++ programming by novice students, based on the works for Java programming. For the effective VTP instances for C++ programming study, it is important to select proper source codes that contain basic grammar concepts to be studied by the students. It is emphasized that C++ greatly differs from Java in the memory managements, such as allocating and de allocating memory manually by the program. C++ supports the memory access using a pointer for implementing fast and efficient programs. These concepts must be covered in the generated VTP instances to understand the advantage in C and C++ programming. In addition, the object-oriented programming concept of C++ programming must be studied.
Based on the abovementioned conditions, we collected 37 source codes from websites and textbooks for basic grammar concepts in C++ programming, and generated VTP instances manually, after analyzing important variables and outputs messages in the codes.
Then, to verify the effectiveness of the generated 37 VTP instances, we assigned them to a total of 46 students in three universities in three countries. They include 17 first-year or second-year undergraduate students in Yangon University in Myanmar, 13 graduate students in Okayama University in Japan, and 16 graduate students in Electronic Engineering Polytechnic Institute of Surabaya University in Indonesia. They are either currently studying or have studied C++ programming in universities. These students were requested to solve them at home using the answer interface for JPLAS by themselves as self-study without tutoring from the teacher. Their solution results show that among the 46 students, 35 students solved all the questions correctly. Only two students had the difficulty in solving them. These students need cares from the teacher at this early programming study stage.
The rest of this paper is organized as follows: Section II review preliminary works related to this study and literature. Section III presents method for value trace problems for C++ programming. Section IV includes the result and discussion based on evaluating them through applications to students. Finally, Section V concludes this paper with future work.

II. PRELIMINARY WORK
In this section, we review our preliminary works to this study. We also discuss some related works in literature.

A. Answer Interface
To solve exercise problems in JPLAS without the Internet connection at home, the web-based offline answer interface has been implemented in [4]. This function supports students to solve VTP instances on a browser, to keep the compatibility with the online JPLAS [5]. The answer marking procedure by string matching is implemented using JavaScript to run on the browser. The function contains the correct answers of the questions, where they are encrypted using SHA256.

B. VTP Generation Procedure
To compose a VTP instance, a teacher needs to prepare a source code, a set of questions, the answer forms (blanks), and the correct answers. To generate a new VTP instance, the following procedure should be complied: 1) Select a source code to be studied by the students from a website or a textbook. 2) Find the important variables and standard output messages in the source code, and make the questions of asking the actual values or messages of them. 3) Insert the standard output statements into the source code to output them by compiling and running it. 4) Collect the correct answers to the questions by running the code and observing the corresponding standard outputs from the code. 5) Make the source code, the questions, and the correct answers into one text file. 6) Run the program in [4] with the text file in e), and generate the HTML/CSS/JavaScript files for the offline answer interface for this VTP instance. 7) Register the generated VTP instance in the assignment to students.

C. Related Works in Literature
Here, we briefly discuss related works in literature as follow.
In [6], Hwang et al. proposed the web-based programming assisted system for cooperation (WPASC) designing learning activity for facilitating cooperative programming learning, and investigated cooperative programming learning behaviors of students and the relationships with learning performances.
In [7], Quinson et al. presented the programmer's learning machine (PLM) as an interactive exerciser aimed at learning programming and algorithms. It targets students in (semi) autonomous settings, using an integrated and graphical environment that provides a short feedback loop. This generic platform also enables teachers to create specific programming micro-worlds that match their teaching goals. PLM provides two main panels to provide information for students to solve exercises.
In [8], Samy et al. developed the intelligent tutoring system called the CPP tutor for the interactive learning environment for students. Students will be able to learn C++ programming and to gain knowledge more quickly and effectively than students using traditional methods of teaching.
In [9], Ihantola et al. presented a systematic literature review for automated assessments of programming assignments, which includes major features and approaches from the pedagogical and technical points of view.
In [10], Jain et al. developed an educational tool for understanding algorithm, and building and learning programming language. This tool provides an innovative and unified graphical user interface for developments of multimedia objects, educational games, and applications. It also provides an innovative method for code generations to enable students to learn the basics of programming languages using drag-n-drop methods of image objects.
In this paper, we generate value trace problems for C++ programming and investigate the effectiveness for novice students to learn basic grammar concepts through code reading by themselves. Students can get learning achievement by reading and understanding the code by using offline answer interface.

III. METHOD FOR VALUE TRACE PROBLEM FOR C++ PROGRAMMING
In this section, we present generations of value trace problems (VTPs) for studying basic grammar concepts in C++ programming.

A. VTP for Basic Grammar Concept
In this paper, we generated 37 VTPs using source codes in websites [11]- [13] and textbook [14] for basic grammar concepts in C++ programming. Table I shows the instance ID, the basic grammar concept, the number of lines in the source code, the number of questions, and the number of answer forms on each VTP instance. Fig. 1 illustrates the answer interface for the example VTP instance at ID=15. This question asks the values of important variables, i and cars [i]. Their correct answers are 0, Volvo, 1, BMW, 2, Ford, 3, and Mazda.

B. Example VTP Instance
A student needs to read the source code carefully to understand it, fill in the forms, and click the "Answer" button.
Then, the form becomes white if the answer is correct, and red otherwise. The student can repeat the answering process until all the answers become correct.

IV. RESULT AND DISCUSSION
In this section, we evaluate the generated 37 VTP instances for C++ programming through applications to a total of 46 students in three universities in three countries, namely, 17 first-year or second-year undergraduate students in Yangon University, Myanmar, 13 graduate students in Okayama University, Japan, and 16 graduate students in Electronic Engineering Polytechnic Institute of Surabaya University, Indonesia. We analyze their solution results.

A. VTP for Basic Grammar Concept
As shown in Table I, the 37 instances contain the total of 126 answer forms (blanks). We asked the students to solve them using the offline answer interface for JPLAS at home. These students have studied basic grammar concepts for C++ programming. The first-year and second-year undergraduate students in Yangon University have studied them for three months, the second students have for at least one year, and the graduate students have for several months at their undergraduate programs.

B. Student Solution Results
First, we analyze the performance of each student in solving the VTP instances. Table II shows the student ID, the number of correctly solved answer forms, the total number of answer submission times to mark the answers, the missing instance ID in solving, and the correct answer rate among the 126 forms.

1) Individual student analysis
Based on the results, most of the students solved all the instances correctly. Particularly, the three students with ID=3, 7, and 19 are excellent, because they solved every VTP instance with only one submission. On the other hand, the 11 students with ID=5, 6, 14, 15, 16, 17, 23, 24, 27, 29 and 30 could not solve some instances. The two students with ID=16 and ID=17 missed to solve one instance, although they show the high answer rate 97% and 96% respectively. Although they should be more careful in solving all the instances, it will be necessary to improve the interface of the answer interface to avoid it. The two students with ID=14 and 15 show the lower correct rate than 90%. These students will need a lot of efforts to improve the understanding of C++ programming.
The average number of submission times was 58.1, which suggests that a student submitted his/her answers for one instance two times on average. The standard deviation (SD) is 16.8, which suggests the large diversity among the students.
The average correct answer rate 98.5% indicates that a student correctly solved 98.5% of the questions on average. Table III shows the distribution of the correct answer rates of the students. This table indicates that 35 students among 46 achieved the 100% rate, and nine did over 90% correct rate. Only two students did under 90% rate, who will need to take time for learning the fundamental C++ programming.  Table IV shows the distribution of the numbers of answer submission times by the students. The three students with ID=3, 7, and 19 correctly solved every instance by submitting the answer only once. They are excellent students and understand C++ programming very well. The nine students with ID=1, 12, 24, 26, 28, 31, 43, 44, and 45 submitted the answers more than 75 times. Among them, the eight students with ID=1, 12, 26, 28, 31, 43, 44, and 45 achieved the over 100% correct rate. Thus, these students were seriously solving the VTP instances by submitting answers many times, and actually reached the correct answers, although they will need more practices in C++ programming.

C. Individual VTP Instance Result
Next, we analyze the solving results of the individual VTP instances by the students. Table V shows the instance ID, the number of students who did not attempt to solve, the total number of answer submissions to mark the answers, and the average correct answer rate among the 46 students.  Table V indicates that in general, the 37 VTP instances are relatively easy for them. The average correct answer rate is 100% for any instance except for the 11 instances with ID=2, 16, 17, 18, 20, 21, 31, 32, 34, 35, and 36. These instances may be relatively difficult for them, where up to 11 students made mistakes in them. Table VI shows the distribution of the correct answer rates of the VTP instances. 26 VTP instances among 37 achieved 100% correct rate. 10 VTP instances achieved over the 90% correct rate. Thus, they are suitable for self-studies of novice International Journal of Information and Education Technology, Vol. 12, No. 7, July 2022 students. The one instance ID=36 achieved less than 90% correct rate. This instance may be difficult for novice students. In future works, we will improve these instances so that more students can try to solve and answer them correctly.  Table VII shows the distribution of the numbers of submission times by the instances. 29 instances among 37 are submitted over 50 times and 4 instances are submitted over 75 times. Students submitted answers for one instance two or three times in average so these instances are suitable for the novice students. Four instances are submitted over 100 times so these instances may be difficult for the novice students so we will improve these instances in future works.  Table VIII shows the grammar concepts of the VTP instances where the average correct rate did not reach 100%. They include data type, array, string, pointer, and object-oriented programming specific concepts, which are generally hard for novice students at studying C++ programming. The teacher should explain them in more comprehensible ways using illustrations or subsidiary tools. Besides, it will be necessary to implement hint functions in the VTP answer interface, to help understanding of these hard concepts, which will be in future works. In this paper, we investigated the effectiveness of the value trace problem (VTP) for self-study of C++ programming through code reading study. We generated 37 VTP instances using simple source codes for basic grammar concepts of C++ programming, and asked 46 students to solve them using the answer interface at home.

4) Analysis of hard instances
By analyzing the answer results of the students, we confirmed the effectiveness of the VTP instances for the novice students. We could detect the understanding levels of them in C++ programming and the hard concepts for them, namely, data type, array, string, pointer, and object-oriented programming specific ones. In general, they can be difficult for any novice student at studying C++ programming. It is recommended that the teacher should explain them in more comprehensible ways using illustrations or subsidiary tools in the programming course. Besides, a hint function to encourage the understanding should be implemented at the answer interface, which will be in our future works.
In future studies, we will study improvements of VTP instances to hard C++ programming concepts, and implement the hint function in the answer interface. We will also continue generating VTP instances for other grammar concepts or programming topics such as libraries, data structure, and algorithms, and will assign them to students in C++ programming courses.

CONFLICT OF INTEREST
Authors point out that no conflict of interest.

AUTHOR CONTRIBUTIONS
Shune Lae Aung conducted the design, implementation and experiments of the proposal and wrote the paper. Nobuo Funabiki gave the idea for this research and supervised whole the activities. Nem Khan Dim and Soe Mya Mya Aye assigned the VTP instances to the students and collected the answers from them. Htoo Htoo Sandi Kyaw assisted writing the paper and analyzing the data.