A Multi-criteria-Based Expert Modeling for Assessing the Textbook Quality

 Abstract —This research considers the application of a multi-criteria approach to assess the quality of textbook content. The issues of determining evaluation criteria and organizing the work of experts are brought to the fore, as these factors directly affect the objectivity and efficiency of the evaluation process. The definition of criteria consisted of reviewing common mistakes and comments on the quality of textbooks, forming a preliminary and final list of criteria. The organization of the work of the experts consisted in the development of a platform with the introduction of a multi-criteria approach. As a result, 39 criteria were formulated and proposed, grouped into 10 main sections such as “Structure and organization”, “Content”, “Didactic aspects”, “Methodological aspects”, “Language level”, “Psychological and psycholinguistic aspects”, “Cultural and Value Aspects”, “ICT Integration”, “Differentiation Aspects”, and “Layout and Design”. The work of experts in the developed expert system is described, which helps in making a decision. The implementation of the multi-criteria approach in this expert system is the main contribution of the research work, which helps to improve the process of evaluating textbooks. The paper presents the results of the survey among 44 experts that helped to identify the most significant criteria which affect the evaluation a textbook. The paper presents the results of the survey among 44 experts that helped to identify the most significant criteria which affect the evaluation a textbook. Moreover, the base of allowed errors was formulated for the following types: for illustrations, QR codes, for texts, for the language of presentation, technical errors, content errors, errors for multimedia objects.

this.Yet one thing remains unchanged-the use of textbooks in teaching.The textbook is the main compass orienting the teacher and the student in the assimilation of new knowledge [1].Moreover, textbooks reflect socio-cultural values, political ideologies that are distinct from the dissemination of knowledge.Indeed, at the level of UNESCO, the development and revision of textbooks and curricula was recognized as a high priority [2,3].In June 2018, the First International Summit was held in London, where 14 countries came to the conclusion that the neglect of textbooks is a gross mistake in public policy and international comparisons reveal important principles of quality [4].As you know, the advantages of quality textbooks are enough.They help standardize learning, provide structure according to the curriculum, can educate both students and teachers, and define the main trajectory.Among the shortcomings of low-quality textbooks are deviations from the direction of study, inconsistency with the needs of students, distortion of content, etc. [5].It is important to emphasize that in the study, textbooks mean a set of educational publications, which include educational and methodological complexes, methodological guides for performing practical and laboratory work, electronic textbooks that accompany digital content, etc.
The review of textbooks is a necessary condition for any effective educational process, which is characterized by the achievement of the general goals set by the education system, as well as the goals and objectives outlined by a well-organized curriculum.
The need for quality textbooks is increasing every day, and new methods of evaluating their quality are emerging.Different countries have different textbook approval systems, but in general they can be divided into four main models: liberal or laissez-faire, centralized, accreditation model and professional assessment model [6].Whereas in the US, the textbook board focuses on textbook content and style, organization of learning activities, assessment strategy, and assessment materials, in the Philippines, the National Book Development Board (NBDB) has a policy of evaluating books based on curriculum coverage competencies, accuracy of content, relevance to society and culture, and correctness of the language used [7].Many developing countries strategically focused on quality education are developing their own criteria for evaluating educational materials.Pasaribu in his article gives a variety of criteria for evaluating the quality of textbooks, in which he highlights: compliance with the curriculum, integration with teaching methods, the function of helping the teacher and the student, the attractiveness of the illustrations used, stimulating creativity, consolidating the values of adults and children [1].
The international literature and practice of various educational systems emphasizes the diversity of approaches to the examination of textbooks.Despite the great variety of types of verbal examination in the form of reports, the most common form of examination is the -structured examination‖ of the most important characteristics of the textbook [8].They are defined at the very beginning and combine the main and most valuable features of an effective textbook.It should be mentioned that this approach may vary, as its structure depends on the concept of examination underlying it.Undoubtedly, the criteria for evaluating textbooks should simultaneously cover the maximum number of development aspects and a more specific formulation of the requirement.This prompted the use of a multi-criteria approach to assess the quality of textbooks in our research.
The next necessity in achieving a quality textbook is the effective organization of the work of the experts who evaluate it.After all, all the processes accompanying the evaluation of the textbook can affect the speed and objectivity of the decisions made.Optimization of this process is impossible without the use of digital technology.There is a problem of workload of experts, which negatively affects the quality of assessment.In addition, often repeated mistakes of the authors complicate the work and slow down the productivity of experts.The fragmentation and heterogeneity of the information processed in the process of reviewing the content of textbooks, the lack of knowledge management tools and a scientifically based quality assessment system shows the unresolved problem of providing experts with digital intellectual services for conducting an examination and making objective decisions, as well as monitoring the examination process.
In connection with the above problems, complex research tasks were set:  firstly, to develop a science-based system for assessing the quality of textbooks based on multi-criteria assessment  secondly, to develop a platform for the effective organization of the work of experts The purpose of this study is to determine the impact of multi-criteria assessment through a system for experts on improving the quality of textbooks.
The study touches upon the issues of the specifics of the implementation of multi-criteria assessment of the quality of textbooks in an expert system and the optimization of the work of experts.

II. LITERATURE REVIEW
Multi-criteria evaluation is an approach to decision evaluation that takes into account several criteria instead of one.In this approach, solutions are evaluated on the basis of several attributes and the solution is considered the best if it has the best performance in all criteria.For example, when assessing the quality of an English textbook for an Indonesian high school, this method has shown its applicability [9].As part of the American Project 2061, which aims to improve science education, the following criteria are used to assess the learning effectiveness of each textbook: providing a sense of purpose, taking account of student ideas, engaging students with relevant phenomena, developing and using scientific ideas, promoting students 'thinking about phenomena, experiences, and knowledge, assessing progress, enhancing the science learning environment [10,11].
The results of a study on the application of the Multi-Criteria Decision Making Method (MCDM) over the past decade are conceptually presented in the work of Taherdoost H. and Madanchian M. Examples of the main elements of the MCDM concept are given, such as alternatives, aggregation, criteria, preferences, decisions, etc.The main categories and various classifications of MCDM are described.According to the ScienceDirect database, from 2012 to 2022.10116 articles used the keywords -multi-criteria decision making‖ and 7619 articles -MCDM‖, respectively.Moreover, the increase in published scientific articles in recent years on the use of MCDM shows that it has become more popular [12].
The multi-criteria assessment method has proven its effectiveness and is widely used in many areas.It helps in making complex decisions with a collaborative approach [13][14][15], justifying and weighing each element of influence on the final conclusion.
In the studies of Biswas and Pamucar multi-criteria analysis was successfully used to decide on the most suitable location for B-schools (Business schools).They identified the factors influencing the decision when choosing an educational institution and identified the main criteria such as location awareness, convenience in traveling, commutation time, connectivity with market, connectivity with the recruiters/industrial zones, availability of the amusement facilities, availability of medical facilities, internet accessibility, environment friendliness, safety, cost of commutation/living, familiarity with the local language [16].In addition, various MCDM methods such as Evaluation based on Distance from Average Solution (EDAS), Additive Ratio Assessment (ARAS), Multi-Attributive Border Approximation Area Comparison (MABAC), Measurement of Alternatives and Ranking according to compromise Solution (MARCOS) and Complex Proportional Assessment (COPRAS) have been successfully used in selecting the most suitable SMD (smart mobile devices) resource for MCC (mobile crowd computing) [17].
The use of a multi-criteria approach to assess the quality of service and website selection problems has proven to be effective.A study covering the period from 2005 to 2016 showed that the most common individual approach to assessing the quality of service of a website and its choice is AHP (the analytic hierarchy process) and the main criterion for its performance is responsiveness [18].Moreover, multi-criteria decision-making models help in evaluating environmental education programs [19], the effectiveness of e-learning (e-learning) [20], and even in evaluating acquired knowledge, abilities, and actitudes at the Energy Markets course [21].
Thus, the applicability of the multicriteria approach in different fields of science prompted us to use it to improve the quality of textbooks through the use of this model in an expert system.

III. METHODS
The study began by identifying the need to improve the quality of textbooks.For this, 77 comments on the quality of International Journal of Information and Education Technology, Vol. 13, No. 9, September 2023 textbooks from various sources, such as news portals, newspapers, electronic media, social networks and letters to higher authorities, were reviewed and analyzed.According to the results of which 23 errors were confirmed.Moreover, many errors occurred several times: 4 misprint errors, 8 technical errors, 9 factual errors, 1 substantive error, 1 mistranslation (Fig. 1).This fact also enhances the relevance in improving the quality of the content of textbooks.An objective assessment of the quality of the content of textbooks is also an urgent need in our realities.
Preliminary identification of errors in the content of textbooks helped determine the main criteria that will be used to develop detailed criteria.
The main stage of the study was the development of detailed criteria for evaluating the quality of textbooks.An analysis of existing criteria helped formulate 78 criteria, which are grouped into 10 main sections (Table I).They were then submitted to the experts for consideration.Implementation of the learning objectives specified in the curriculum (knowledge, skills and application) 8 Availability of basic knowledge (themes and subtopics provided for by the curriculum) 9 Logical sequence in the content of the textbook (logicality and interconnection of chapters, sections, paragraphs and topics) 10 Using the previous knowledge of students 11  Didactic aspects 27 The presence in the textbook of materials on applied, practice-oriented strategies for active and interactive learning 28 The presence in the textbook of various tasks and tasks for the development of critical thinking and analytical skills 29 The presence in the textbook of various tasks and tasks for the development of creative thinking 30 The presence in the textbook of personality-oriented tasks and tasks Methodological aspects 31 Orientation of tasks, assignments and exercises to achieve learning outcomes (LO) 32 Clarity in the wording of tasks and assignments 33 Implementation of the principle -from simple to complex‖.Gradation of educational materials according to the level of complexity 34 Relevance, balance of tasks, assignments and exercises in quantity, compliance with learning objectives

35
The focus of the methodological apparatus and the assessment system on the implementation of learning outcomes (knowledge, skills and application) 36 Formation and development of functional literacy 37 Tasks that motivate and promote the active participation of students in the educational and cognitive process 38 The presence in the textbook of graphs, tables, diagrams that facilitate the perception of educational content 39 Tasks, tasks that provide motivation for independent educational, cognitive and research activities 40 The presence of tasks aimed at developing all levels of thinking skills according to Bloom's taxonomy 41 Opportunities for Formative Assessment 42 The presence of varieties of tasks, assignments and exercises 43 Tasks, tasks that provide motivation for joint educational and cognitive activities 44 Tasks and assignments that contribute to the use of various combinations of joint educational and cognitive activities (individual, pair, group, etc.) 45 Providing Self-Assessment Opportunities 46 Tasks that contribute to the development of communication skills (individual, student-student, student-teacher, students) Language level 47 Correspondence of the language of presentation to age and cognitive level 48 Unity in the interpretation of terms and definitions 49 Spelling Compliance 50 Accessibility and clarity of language Psychological and psycholinguistic aspects 51 Compliance of the content of the textbook with the age and psychological characteristics of students 52 The focus of the content of the textbook on the development of educational interests, intellectual and creative abilities of students 53 The focus of the content of the textbook on the development of students' motivation for in-depth study of educational materials 54 Correspondence of the visual and illustrative material of the textbook with the age, psycho-logical characteristics of students, ethical and aesthetic ideas 55 Correspondence of the text, sentence structure and phrase combinations to the age of students, the tasks of better understanding the content 56 Correspondence of new concepts, definitions and terms to the age and experience of students 57 The presence of a variety of tasks and exercises for the development of general and practice-oriented types of speech activity Cultural and value aspects

58
The focus of the textbook on the basic values defined in the state standards, the focus of topics and tasks on the implementation of educational tasks 59 Representation in the content of educational materials and illustrations about the natural and historical features of the country 60 Representation in the content of educational materials and illustrations about the national-cultural and family values of the country 61 Representation in the content of educational materials and illustrations about state priorities and socio-economic, technological achievements of the country 62 Representation in the content of educational materials and illustrations about universal values 63 Attention to issues of social equality, respect for ethnic and cultural groups and religions, tolerant attitude towards people with disabilities ICT integration 64 Opportunities for independent and collaborative work using ICT competencies 65 Integration of digital services and tools with the textbook 66 Motivating students to complete educational tasks (including independently) using ICT 67 Availability of a list of official sources of educational information for students (links) Aspects of differentiation 68 Providing support to students, taking into account different levels of knowledge and cognitive abilities 69 Promote educational materials to ensure equality in education and avoid social differentiation and infringement in educational content Layout and design 70 Consistency and convenience of the structure of the organization of educational materials on the pages 71 Compliance with sanitary and hygienic requirements for the weight and format (size) of the textbook 72 Readability and convenience of fonts and line spacing 73 Ease of navigation (table of contents, indexes and symbols) 74 The attractiveness of the textbook appearance (graphic organizers, images and illustrations, colors, etc.) 75 Ensuring the harmony of text and illustrative materials based on design principles (balance, hierarchy, contrast, space, scale and combination) 76 Technical quality of visual elements, use of colors 77 Content and visual difference between types of texts (main, explanatory and additional) 78 Headings and other navigational support (titles, headings, instructions, typographical notes) As you know, a prerequisite for the effectiveness of multi-criteria assessment is the approbation and feedback of experts who evaluate textbooks in practice.
Approbation of the proposed initial list of criteria showed that due to the non-universality of textbooks, that is, different types of textbooks with the requirements for them are not always satisfied and need to be improved through the grouping of criteria with the necessary descriptions.
Therefore, in order to determine the importance of each proposed criterion as a qualitative feature and assign a weight share, a survey was conducted of 44 experts who have more than 5 years of experience as an expert.According to the results of the survey, the proposed criteria were improved and reformulated to adapt to the developed expert system.
At the next stage of the study, work was carried out to develop an expert system for evaluating the quality of textbook content (Fig. 2).An expert system embodies the experience of an expert in the intelligence of a computer, is able to generate recommendations, make an intelligent decision on a specific operation/function based on the knowledge base.
The expert system database was powered by the advanced, enterprise-class open-source relational database system PostgreSQL.It extends the SQL language, combined with many features to securely store and scale the most complex data work-loads.As an ORM (Object Relation Mapping) tool, Django ORM was used, which interacts with databases using high-level Python methods.When creating the data-base, the code-first approach was applied (Fig. 3), which allows you to manage and focus on the subject area of the class creation platform.In the course of constant refinement of the back-end part of the platform, a normalization approach was used for the correct functioning of the database.Normalization aims to eliminate redundant (repetitive) data and provide logical data storage.
The developed expert system includes many modules that automate and optimize the work of experts, positively affecting their productivity.Nevertheless, a full assessment of the content component of textbooks is impossible without the physical participation of experts who embody the approach of human-computer interaction, taking into account the contextual, semantic load.Therefore, the expert system pro-vides experts with digital intellectual services for conducting examination and making objective decisions, as well as monitoring the examination process.A feature is the introduction of the above proposed multi-criteria assessment based on a multivariate analysis of the examination process with the selection of the main vectors and components of the expert assessment.
In general, the expert system contributes to a deeper understanding of inductive reasoning and machine learning, allows you to implement the organization and presentation of data, interpret, search and apply expertise data to gain knowledge, make decisions and, ultimately, make predictions, avoiding bias.The snippet below shows the detection of errors and the definition of error types (Fig. 4).At this stage, experts look through textbooks and, if errors are found, select the necessary fragments and select the type of error from the list proposed by the system by entering their comments.All entered comments are displayed on the right side with a page indication.In the future, all comments and errors will be included in the expert decision.
Depending on the type of textbook, each section is assigned a weight share, which helps to prioritize the assessment criteria (Fig. 5).The ability of an expert to justify his choice with the help of an appropriate commentary, example, or scientific argument, where the expert considers it necessary.Brief written notes are specific and concise and in most cases are the main benefit of a particular criterion.Experts do not have the opportunity to score at random, as they may have to justify their choice.So, in the case of setting less than three points according to the evaluation criterion, in the -Comment‖ column, the experts briefly and concisely justify the reasons for the decrease.In the code of errors and shortcomings, the reasons for the decrease are indicated in more detail.The methods used in this study led to the formation of clear criteria with the necessary descriptors based on the results of a survey of experts and the implementation of the development of an expert system for evaluating the quality of textbook content.

IV. RESULTS AND DISCUSSIONS
As a result of the research work, a positive effect was obtained from the implementation of a multi-criteria assessment of the quality of a textbook in an expert system.This is confirmed by the fulfillment of research tasks and the achievement of the goal.
A detailed survey of 44 experts made it possible to identify more significant criteria that help in evaluating and determining the weight share of each criterion in the expert system.
The first question was supposed to identify the most important aspect of the text-book evaluation from the proposed 10 sections.Experts rated on a scale from 1 to 10, where 1 is the lowest importance, 10 is the highest importance (Fig. 6).As a result, it turned out that the most important sections, according to experts, were -Content‖ and -Methodological aspects‖ of the textbook.The maximum number, that is, 29 respondents out of 44, rated these sections at 10 points.
The following questions helped uncover the criteria for each section.
The questions in Section 1 -Structure and organization‖ were intended to identify the significance of each criterion, on a scale from 0 to 6. Respondents rated 1 for the lowest importance, 6 for the highest importance, and 0 if the criterion is not needed or not measurable (Fig. 7).
According to the results of the survey, 26 experts rated the -Table of Contents‖ at 6 points, thereby determining this criterion as the most significant for this section.The second most important criterion was -Learning objectives, description at the beginning of each chapter‖, which was rated by 24 experts at 6 points.The -Applications‖ criterion received the 11th lowest number of top scores.The results for Section 2 -Content‖ showed that the criterion -Implementation of the learning objectives specified in the curriculum (knowledge, skills and application)‖ turned out to be the most significant, because 33 experts rated it with the maximum score.The criterion -Clarification of common misconceptions, erroneous versions‖ turned out to be the least significant, since only 7 experts voted for it (Table II).Availability of basic knowledge (themes and subtopics provided for by the curriculum) 31

Reliability of information. Correspondence of the presentation of the content to scientific and problematic approaches 31
Logical sequence in the content of the textbook (logicality and interconnection of chapters, sections, paragraphs and topics) 29

Relevance of information 28
Compliance of the level of complexity of the content with the age characteristics and cognitive abilities of students 26 Correspondence of visual elements (graphics, pictures, maps, etc.) to the educational content 23 Using Appropriate Units and Metric Symbols 21 Adaptation of the content to the conditions of real life and activity 21 The presence of correct and understandable labels for visual elements (graphics, pictures, maps, etc.) 20

Intra-subject communication in the content of the textbook 19
Using the previous knowledge of students 18 Balance between the volume and depth of the stated subject content 17 The presence of new (non-standard) situations / examples in the textbook 16 Specify the resources needed to complete the learning tasks 14 Interdisciplinary integration of educational content 14 A variety of forms of presentation of educational content in the main, additional and explanatory texts 14 Availability of additional literature for reading 9 The degree of localization of the context (local history aspects) 8 Clarification of common misconceptions, erroneous versions 7 The results in Section 3 -Didactic Aspects‖ showed that the criteria -The presence in the textbook of various tasks and tasks for the development of critical thinking and analytical skills‖ and -The presence in the textbook of various tasks and tasks for the development of creative thinking‖ are of the greatest importance (Fig. 8).
The criterion -Orientation of tasks, assignments and exercises to achieve learning outcomes (LO)‖, which refers to Section 4 -Methodological Aspects‖, turned out to be the most significant.For this criterion, 31 experts gave the maximum score.The criterion -Tasks that promote the development of communication skills (individual, student-student, student-teacher, students)‖ received only 11 maximum points, which showed the least significance of all proposed (Table III).Clarity in the wording of tasks and assignments 30 Implementation of the principle -from simple to complex‖.Gradation of educational materials according to the level of complexity 30 Relevance, balance of tasks, assignments and exercises in quantity, compliance with learning objectives 26 The focus of the methodological apparatus and the assessment system on the implementation of learning outcomes (knowledge, skills and application) 26

Formation and development of functional literacy 26
Tasks that motivate and promote the active participation of students in the educational and cognitive process 25 The presence in the textbook of graphs, tables, diagrams that facilitate the perception of educational content 24 Tasks, tasks that provide motivation for independent educational, cognitive and re-search activities 23 The presence of tasks aimed at developing all levels of thinking skills according to Bloom's taxonomy 22 Opportunities for Formative Assessment 18 The presence of varieties of tasks, assignments and exercises 17 Tasks, tasks that provide motivation for joint educational and cognitive activities 16 Tasks and assignments that contribute to the use of various combinations of joint educational and cognitive activities (individual, pair, group, etc.) 14 Providing Self-Assessment Opportunities 14 Tasks that contribute to the development of communication skills (individual, student-student, student-teacher, students) 11 In section 5 -Language level‖, the criterion -Accessibility and clarity of presentation‖ turned out to be the most important, because it received 38 highest votes of experts (Fig. 9).In Section 6 -Psychological and psycholinguistic aspects‖, the criterion -Compliance of the visual and illustrative material of the textbook with the age, psychological characteristics of students, ethical and aesthetic ideas‖ received 34 points, which is the highest number of maximum points (Fig. 10).The results of the survey on Section 7 -Cultural and Value Aspects‖ showed that the criterion -The focus of the textbook on the basic values defined in the State Compulsory Standard of Education (SCSE), the focus of topics and tasks on the implementation of educational tasks‖ is of significant importance (Fig. 11).In Section 8 -Integration of ICT‖, the criterion -Motivation of students to complete educational tasks (including independently) using ICT‖ turned out to be the leading one with 22 highest expert scores (Fig. 12).In Section 9 -Aspects of differentiation‖, the criterion -Providing support to students taking into account different levels of knowledge and cognitive abilities‖ received 34 maximum votes (Fig. 13).In the last Section 10 -Layout and design‖, the criteria -Logic and convenience of the structure of the organization of educational materials on the pages‖ and -Compliance with sanitary and hygienic requirements for the weight and format (size) of the textbook‖ turned out to be the most important with 29 expert votes (Table IV).Compliance with sanitary and hygienic requirements for the weight and format (size) of the textbook 29 Readability and convenience of fonts and line spacing 23 Ease of navigation (table of contents, indexes and symbols) 22 The attractiveness of the textbook appearance (graphic organizers, images and illustrations, colors, etc.) 21 Ensuring the harmony of text and illustrative materials based on design principles (balance, hierarchy, contrast, space, scale and combination) 19 Technical quality of visual elements, use of colors 19 Content and visual difference between types of texts (main, explanatory and addition-al) 18 Headings and other navigational support (titles, headings, instructions, typographical notes) 17 In general, the results of the survey showed a positive trend, since, according to experts, many of the proposed criteria are very important.Moreover, the experts identified the most significant criteria for each section, which helped to approve the final 39 criteria by grouping them into 10 sections (Table V).Explanations for each criterion were reflected in the descriptors, which helped the experts in the assessment.
The implementation of multi-criteria evaluation in the developed expert system has optimized the work of experts, adding clarity and efficiency in completing tasks.Using the developed multi-criteria approach in the expert system, the experts were able to fully assess the quality of the textbook content.The expert system was provided with an advanced database where the Code-First approach was applied.As a result of applying the Code-First approach, the workflow looked like this (Fig. 14).Moreover, applying database normalization yielded the following results:  freeing a set of relationships from unwanted insert, update, and delete dependencies  reduced the need to restructure the set of relationships as new data types are introduced and thus the lifespan of application programs is increased  the relational model has become more informative for requests from the back-end  relation sets became neutral with respect to query statistics The textbooks were evaluated by experts on each criterion using a 4-level assessment scale (Table VI).In the case of setting less than 3 points according to the evaluation criterion, in the -Comment‖ column, the experts briefly and concisely substantiated the reasons for the decrease.In the code of errors and shortcomings, the reasons for the decrease were indicated in more detail.An indicative result of the optimization of the work of experts was a pre-formulated base of errors that can be selected for filling in -Comments‖ (Table VII).The final stage is a general analytical conclusion, which serves as the basis for an expert decision.Experts are invited to express in detail their opinion on educational publications in a descriptive form, make suggestions for its improvement and justify their expert decision in detail.
Processing of assigned evaluation points:  the average score of each of 10 sections is determined;  The average score of the educational publication is determined in 10 sections.The resulting number is not rounded.
In the case of an examination of an educational publication by 3 (three) experts, based on their average marks, an arithmetic mean is determined, which is the overall assessment of the expert group for this publication.
Making expert decisions on textbooks: 1) with an average score of 0 to 1.3 points, the textbook is not recommended for use in educational institutions 2) with an average score of 1.4 to 2.7 points, the textbook needs to be improved 3) with an average score of 2.8 to 3 points, the textbook is recommended for use in educational institutions.The data obtained indicate that the original version of the criteria for evaluating textbooks was less effective due to the too large and unstructured enumeration of criteria that do not take into account the characteristics of the types of textbooks, which leads to unsatisfactory indicators for several criteria in practice.Since the multi-criteria approach calculates the average score for all the proposed criteria, the presence of extra criteria can negatively affect the overall result.
Despite this, listing the maximum number of criteria helped to determine the main criteria and offer a new version in which they are grouped by semantic content and supplemented with descriptors, which increases the versatility of its application to different types of textbooks.
The results of the expert survey helped to reveal the importance of certain aspects of textbook evaluation, the significance of criteria, to pay special attention to the content, to highlight the didactic and methodological components, to assess the clarity of presentation, the cultural and value orientation of textbooks, etc.
All this helped to adapt the proposed criteria to the developed expert system for evaluating textbooks, which optimizes the work of experts, simplifying and automating the identification of frequently made mistakes in textbooks.Moreover, the introduction of a weight share for each criterion in the expert system helps to determine its significance based on the type of textbook being evaluated, which increases the objectivity of expert decision-making.

VI. CONCLUSION
Nowadays, textbooks do not lose their relevance due to the value of their content and the definition of the main guidelines in learning.This raises the requirements for the quality of textbooks and raises questions about the objectivity of expert evaluation, taking into account comprehensive criteria.In addition, the workload of the experts themselves is an important factor affecting the quality of the examination.
The conducted study proposes the use of a multi-criteria approach to assess the quality of a textbook with the maximum coverage of criteria.As a result of the study, 39 criteria were formulated, grouped into 10 sections.Each criterion contains descriptors that help experts in their work.Moreover, with the help of a survey of experts, the most significant criteria for each section were determined for assigning a priority weight share.The implementation of the multi-criteria approach in the expert system helped the experts more objectively and quickly identify the mistakes made, comment and justify their decisions on evaluating the quality of textbooks.
The implementation of the multi-criteria approach in this expert system will allow accumulating of multi-criteria expert data and expand the knowledge base.The accumulation of a sufficient volume of data sets for evaluating the quality of textbooks opens up the prospect of researching the use of artificial intelligence methods in evaluating the quality of textbooks and digital content, thereby expanding the intellectual capabilities of an expert and improving the quality of evaluating a textbook and educational content in general.The present work may be further extended using uncertainty based analysis in a fuzzy or rough environment with a group-decision making set up.
Clarification of common misconceptions, erroneous versions 12 The presence of new (non-standard) situations / examples in the textbook 13 The degree of localization of the context (local history aspects) 14 Reliability of information.Correspondence of the presentation of the content to scientific and problematic approaches 15 Relevance of information 16 Using Appropriate Units and Metric Symbols 17 Specify the resources needed to complete the learning tasks 18 Adaptation of the content to the conditions of real life and activity 19 Balance between the volume and depth of the stated subject content 20 Compliance of the level of complexity of the content with the age characteristics and cognitive abilities of students 21 Intra-subject communication in the content of the textbook 22 Interdisciplinary integration of educational content 23 A variety of forms of presentation of educational content in the main, additional and explanatory texts 24 Correspondence of visual elements (graphics, pictures, maps, etc.) to the educational content 25 The presence of correct and understandable labels for visual elements (graphics, pictures, maps, etc.) 26 Availability of additional literature for reading

Fig. 6 .
Fig.6.Rating the importance of each section of textbook evaluation aspects.

TABLE I :
CRITERIA PROPOSED FOR ASSESSING THE QUALITY OF TEXTBOOKS № CriteriaStructure and organization of the textbook 1 Table of contents 2 Learning objectives described at the beginning of each chapter 3 Introduction to each chapter 4 Additional and explanatory texts 5 Applications 6 List of sources used Content 7

TABLE II : Criteria ScoreImplementation of the learning objectives specified in the curriculum (knowledge, skills and application) 33

TABLE III :


TABLE IV :
SECTION 10 -LAYOUT AND DESIGN‖

TABLE VII :
TYPES OF ERRORS ALLOWED