Research on Evaluation Methods of Scientific Research Performance

Abstract—Scientific research is a systematic creativity work for enhancing knowledge and inventing new technologies. Thus, it is of great significance to evaluate scientific research performance from the perspective of knowledge flow. This paper presents an evaluation method of scientific research performance which matches the scientific research mission about using existing knowledge to create new knowledge, and provides a new perspective on multi-angle evaluation of scientific research performance. The knowledge flow network formed by citation relationships between literatures will be constructed firstly. Based on this citation network, the process in which literature knowledge is acquired, absorbed, rationally criticized and new knowledge is created will be as an evaluation criterion. Then, both knowledge utilization rate indicator and knowledge net utilization rate indicator will be established.


I. INTRODUCTION
Scientific researches show that scientific knowledge has obvious accumulation and inheritance. Any new discipline or technology is derived on the basis of the original discipline or technology. It is the development of the original discipline or technology. [1] This process of knowledge evolution and innovation is accompanied by the knowledge flows, which originate from the knowledge potential difference between the knowledge exporter and the knowledge receiver. From the evolutionary perspective of innovation, research collaboration and external knowledge flows are seen as important catalyzers for acquiring new capabilities for innovative organizations which cannot rely only on internal knowledge base. [2] Knowledge flows occur in any social, fluid environment where learning and collaboration can take place and are quickly becoming one of the most crucial sources of value creation. [3] Scientific papers are an important knowledge carrier, and the citation between papers Manuscript received June 14, 2019; revised February 29, 2020. This work was supported in part by Science  is one of the basic forms of knowledge flow. Junsheng Zhang et al. studied the citation-based knowledge flows among research elements such as researcher, paper, publication venue and research topic. [4] The knowledge export through citation reflects the true value of scientific research. [5] Thus it is of great significance to evaluate the scientific research performance from the perspective of knowledge flow generated by the paper citation network.
Global researchers have carried out many research works on scientific performance evaluation based on citation knowledge flow network. WANG Liang et al. analyzed the process and mechanism of knowledge flow from the perspective of citation network. [6] Fragkiadaki et al. presented f-value indicator that measured the importance of a research article by taking into account all citations, directly and indirectly, which was calculated by an algorithm. [7] Walker et al. introduced the PageRank link network analysis method into the citation network, and proposed a scientific publication evaluation method based on PageRank. [8] A model was also built to measure the academic impact of papers in three dimensions, which were the input of knowledge, the output of knowledge and the structure of knowledge flow network. [9] Under the analysis on knowledge flow in hybrid documents co-citation network, GAO Jiping et al. makes concordance between network properties, cited frequency, betweenness centrality, and cited half-life, and knowledge attribute, knowledge impact, knowledge control and knowledge vitality, in order to reveal the high-quality documents. [10] Saeed-Ul Hassan introduced International Scholarly Impact of Scientific Research (ISISR), a new quantitative measure of international scholarly impact of countries by using bibliometric techniques based on publication and citation data. [11] This measure shows the ability of a country to compete for the citations made by the papers authored by researchers from outside the country in a given subject area. Wang Xuemei et al. developed two new bibliometric indices-the Citation Flow Index (CFI) and the Normalized Citation Flow Index (NCFI)-to measure knowledge flows based on scientific literature citations. [12] Hai Zhuge proposed a knowledge flow model for peer-to-peer knowledge sharing and management in cooperative teams, which included the knowledge flow concepts, the knowledge flow rules and methods, the knowledge flow process model, and the knowledge flow engine. [13] In general, the current researches focused on using citation networks to study the theory of knowledge flow, flow mechanism, stage characteristics, evolution models and evaluation of academic papers. However, the researches on evaluating scientific research performance based on knowledge flow were still weak. These scientific performance evaluation indicators analyzed the proportion of the utilization of a research subject by other research subjects relative to the total utilization situation, and did not consider the research body's own paper output volume. In this paper, we established knowledge utilization rate indicator and knowledge net utilization rate indicator to evaluate the scientific research performance.

A. Measuring Indicators for Knowledge Flows
Based on the publication numbers, citation frequency, publication years of papers, the measuring indicators of knowledge flows in knowledge flow networks were constructed. These indicators including knowledge flow direction, knowledge flow intensity, and knowledge flow speed, and knowledge flow breadth, whose definitions were detailed in Table I.

B. Evaluation Indicators of Scientific Research Performance for Countries
Knowledge utilization rate (KUR) is the ratio of citation frequencies of a given country cited by other countries to the total number of papers from X, as shown in Eq. 1.
wherei and j are the serial numbers of a given country, m is the serial number of the published papers for a given country,  is the citation frequency, B is the number of papers, t is the publication year, i C is the total number of the published papers for a given country. Knowledge net utilization rate (KNUR) is the ratio of the difference between the citation frequencies of a given country cited by other countries and the citation frequencies of a given country citing other countries to the total number of papers from X, as shown in Eq. 2.
wherei and j are the serial numbers of a given country, m and n are the serial number of the published papers for a given country,  is the citation frequency, B is the number of papers, t is the publication year, i C is the total number of the published papers for a given country.

A. Data Source
Elsevier's Scopus citation database was used as data source. Using "quantum comput*" OR "Quantum process*" OR "quantum circuit" OR "Quantum chip*" OR "qubit*" OR "quantum algorithm*" OR "quantum simulat*" as the subject words, 24972 papers about quantum computing were retrieved from the Scopus database during the time period 1997-2016. These papers are research articles and review articles.

B. Research Methods and Visualization Tool
A variety of quantitative research methods were used in this study, mainly including statistical analysis, comparative analysis, social network analysis, visualization analysis and other bibliometrics research methods. The scientific computing software MATLAB was used for the construction of citation data and the calculation of evaluation indicators of scientific research performance for countries. EXCEL was used for statistical analysis of data and graphics drawing. Ucinet was used to generate network data from matrix data. Gephi was used for visualizing network data.

A. Number of Quantum Computing Publications
Between 1997 and 2016, the number of global quantum computing publications had shown a general increase in volatility, as illustrated in Fig. 1.

B. Knowledge Flow Network
The knowledge flow network between Top10 countries in the number of global quantum computing publications was shown in Fig. 3. The largest knowledge flows came from United States to Germany, China, and United Kingdom, followed by United States to France, Japan, Spain, and Canada. In addition, the United States, Germany, China, and United Kingdom had strongly internal knowledge flows.

C. Knowledge Utilization Rate
The Top 10 citing countries and corresponding citation rates for USA, UK, Germany, and China in quantum computing every five years from 2001-2016 was shown in Fig. 4. It can be seen that the quantum computing papers in United States, United Kingdom, Germany, and China had the largest self-citation rates. Quantum computing papers from United States were also highly cited by China, Germany, United Kingdom, and Japan. Among them, the citation rates of China and Germany increased, and the citation rates of the United Kingdom and Japan had not changed much. Quantum computing papers from United Kingdom were also highly cited by United States, China, and Germany. The citation rate from United States declined generally, and the citation rate from China was relatively stable. Quantum computing papers from Germany were also highly cited by United States, United Kingdom, and China. The cited rates from United States had small fluctuations. The cited rates from United Kingdom and China both fell first and then rose. Quantum computing papers from China were also highly cited by United States and India, and both fell first and then rose. The citing rates of Top10 countries for China were significantly lower than those of other countries. In order to more clearly show the use of knowledge among countries and consider the number of quantum computing papers of each country, we investigated knowledge utilization rate. Fig. 5 showed the knowledge utilization rate for USA, UK, Germany, and China during 1997-2016. The United States has the highest knowledge utilization rate. Most of the years between 1997 and 2016 were more than 2. The knowledge flow rates of the United States relative to China and Germany was higher than those to other countries. It reached the highest value of 1.06 in 2010, indicating that the performance of scientific research in United States was the best. UK's own knowledge utilization rate fluctuated International Journal of Information and Education Technology, Vol. 10, No. 4, April 2020 between 1-2 in most of the years. The knowledge flow rates of UK relative to USA were significantly higher than those to other countries, followed by China and Germany. Germany's own knowledge utilization rates generally increased with a significant fluctuation, reaching a maximum value of 2.52 in 2004. The knowledge flow rates of Germany relative to USA were significantly higher than those to other countries, followed by United Kingdom and China. China's own knowledge utilization rate was mostly below 1.5, and the knowledge flow rates of China relative to other countries were relatively low, indicating that the performance of scientific research in China was still far from the other three countries. Since the knowledge flow speed is the knowledge flow intensity divided by the paper age and the knowledge flow intensity was equal to the knowledge utilization rate, Fig.  6 and Fig. 5 had the same trend, but the specific values were different.  International Journal of Information and Education Technology, Vol. 10, No. 4, April 2020 There is both the inflow of knowledge and the outflow of knowledge between countries. In order to consider these two situations comprehensively, we had studied the net utilization of knowledge for countries, as shown in Fig. 7. The knowledge net utilization rate of the United States relative to United Kingdom, Germany and China was positive since 1999, indicating that United States was a knowledge exporting country compared to these three countries. And United States had the highest knowledge output to China, followed by Germany and United Kingdom. The knowledge net utilization rates of the United Kingdom relative to the United States were negative for most of the years, indicating that the United Kingdom was basically a knowledge importing country relative to the United States. The knowledge net utilization rate of the UK relative to Germany fluctuated between positive and negative values, and the knowledge net utilization rate of the UK relative to China was positive for most of the years, indicating that the UK was basically a knowledge exporting country relative to China. The knowledge net utilization rate of Germany relative to China was positive for most of the years, indicating that Germany was basically knowledge exporting country relative to China. The knowledge net utilization rates of China relative to the United States, the United Kingdom, and Germany were negative for most of the years, indicating that China was a knowledge importing country. And China's scientific research performance had a certain gap compared with these three countries.

E. Knowledge Flow Breadth
The knowledge flow breadth was shown in Fig. 8. It can be seen that the knowledge flow breadth of the United States was the largest during 1997-2016 and both the UK and Germany increased with a fluctuation, and China showed a linear growth.

V. CONCLUSION
We presented an evaluation method of scientific research performance based on citation knowledge flows, and established knowledge utilization rate indicator and knowledge net utilization rate indicator. Then, a case study was presented to illustrate the use of our proposed measuring indicators in the subject area on quantum computing during 1997-2016. The results shown that knowledge utilization rate indicator and knowledge net utilization rate indicator could reflect the impact of country's own paper output numbers and distinguish scientific research performance of two countries with equal cites per document. These indicators could also be of great importance for other evaluation methods of scientific research performance on institutions and researchers.