decision support system design thesis

Repository KITopen

Design of data-driven decision support systems for business process standardization, abstract (englisch):.

• KITopen-Download
• Statistiken

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Comput Intell Neurosci
• v.2021; 2021

Model Construction and Research on Decision Support System for Education Management Based on Data Mining

Weidan wang.

School of Political and Public Management, Zhengzhou University, Zhengzhou, Henan 450000, China

Associated Data

The data used to support the findings of this study are available from the corresponding author upon request.

Based on data mining technology, this paper applies a combination of theoretical and practical approaches to systematically describe the background and basic concepts related to the generation of data mining-related technologies. The classical data mining process is analyzed in depth and in detail, and the method of building a decision support system for education management based on the B/S model is studied. Not only are the data mining techniques applied to this system, but also the decision tree model with the improved ID3 algorithm is implemented in this thesis, which is further applied to the educational management decision support system of this topic. The load of the client computer is reduced, and the client computer only needs to run a small part of the program. This paper focuses on the following aspects: the overall planning of the educational management decision support system based on data mining technology. From the actual educational management work, we analyze the requirements and design each functional module of this system in detail, applying the system functional structure diagram and functional use case diagram to represent the functional structure of the system and using flow charts to illustrate the workflow of the system as a whole and in parts. The logical structure design, entity-relationship design, and physical model design of the database have been carried out. To improve the efficiency of the system, the ID3 algorithm was improved on this basis to reduce the time complexity of its operation, improve the efficiency of the system operation, and achieve the goal of assessing and predicting the teaching quality of teachers. The development and design of this system provide an efficient, convenient, scientific, and reliable system tool to reduce the workload of education administrators and, more importantly, to make reasonable and effective use of the large amount of data generated in the management, and data mining techniques are used to extract valuable and potential information from these data, which can be more scientific and efficient for the teaching of teachers and students. It can provide reliable, referenceable, and valuable information for managers to make assessments and decisions.

1. Introduction

In the current context of rapid economic growth and rising urbanization, education resources, as one of the important social public resources, are lagging in terms of technical means of management, which poses a negative impact on the deepening of education reform and the optimal allocation of education resources, thus making the development of these tasks face serious challenges [ 1 ]. With the continuous progress of information technology, information technology has been widely used in the field of education management at home and abroad to improve their management level [ 2 ]. Although the educational information management systems currently in use are also based on database construction, such systems can only achieve simple analysis and management of data, and the decision-making function is relatively weak. The system can collect education-related data and conduct simple analysis and statistics on the collected data, but its functions are relatively weak or even absent when it comes to the comprehensive analysis of education management. Many developed countries in the world have established a complete, high-tech, and scientific education management decision support system, which has a huge guiding role in the economic development of their national systems. Many colleges and universities have also established educational management decision support systems that are suitable for their own development in local area networks and wide area networks [ 3 ]. At the same time, the use of data mining technology in education management decision-making has also received more and more attention from universities, and better application of data mining to provide services for education management and decision-making has become a new issue facing education [ 4 ].

For quite a long time in the past, educational decision-making was based on leadership intuition as well as relevant experience or even relied on social trends, which is obviously not in line with scientific decision theory, and therefore became an important criticism content for educational research staff. Scientific decision-making is inseparable from the necessary data support, but, for a long time in the field of education, a lot of data helpful for decision-making is often scattered on the desks and beds of faculty members [ 5 ]. Therefore, strengthening the construction of information technology systems, centralizing and unifying the management of large amounts of data, and then using the corresponding algorithms to analyze them will enable decision-making managers to quickly obtain a large amount of data support, helping them to more accurately understand the strengths and weaknesses of the current educational reality to obtain scientifically correct decisions. Data mining technology can analyze large and complex datasets efficiently and quickly and can find out the information hidden behind the data and the correlations, trends, and directions among the data [ 6 ]. If data mining technology is adopted in the education management system, it has important practical significance for the long-term development of education management. We are still continuing to study improved decision tree algorithms, and corresponding improvements have been made to the C4.5 algorithm from different perspectives. Among them are the time-consuming improvements for the C4.5 algorithm to process continuous attributes, using mathematical methods. The price is infinitesimal to improve the calculation efficiency of the information gain rate and so on [ 7 ].

The study conducted requirements analysis and detailed design of each functional module for this system and applied system functional structure diagram and functional use case diagram to describe the functional structure of the system and flow chart to illustrate the workflow of the system as a whole and in parts. The logical structure design, entity-relationship design, and physical model design were carried out for the database. Then the traditional ID3 algorithm was applied to the teaching quality assessment subsystem for analysis. To improve the efficiency of the system, the ID3 algorithm was optimized and improved on this basis to reduce the time complexity of its operation and improve the system operation efficiency. The goal of assessing and predicting teachers' teaching quality is achieved. This paper is divided into 5 sections, and the structure is arranged as follows: Section 2 discusses related work, the current state of research, and the application of classical algorithms, which lays the foundation for the research use in the subsequent chapters. Section 3 is devoted to research on data mining-based decision support system for educational management. Section 4 is devoted to results analysis. Section 5 gives the conclusion. Data mining techniques are applied to educational management decision support system, relevant teacher information data, student information data, and social background information data are collected, and then the data are preprocessed; data mining algorithms are used to build models, test models, and apply models to help educators understand students' learning characteristics as much as possible, and the factors affecting teaching quality are analyzed. The value of the data related to students' learning information is maximized, and the role of educational data mining is truly brought into play.

2. Related Work

With the acceleration of campus digitalization in recent years, many universities have developed information management systems on their own or jointly with other universities to realize paperless offices and improve the scientific and high efficiency of daily work. The use of this system has accumulated a large amount of data, but colleges and universities have only made some inquiries, statistics, and reserves on these data and have not made full use of them. To make better use of these data and discover hidden valuable information, some universities have started to use data mining techniques to study, analyze, and solve problems. Therefore data mining technology will get more attention and application in the field of education [ 8 ]. Rodrigues et al. studied the application of data mining in the analysis of college entrance examination admission data through association rules and decision tree classification in the design of the college entrance examination information analysis system. Useful knowledge is mined for educational institutions, candidates, and colleges and universities [ 9 ]. Ghorbani and Ghousi conducted an in-depth study on educational management and data mining techniques in colleges and universities, using data mining related algorithms, and integrated a simple voting strategy through the study of simple improvement method of Apriori algorithm and ID3 algorithm, which has improved the educational management of students in colleges and universities and provided valuable guidance for students' training [ 10 ]. Farouk and Zhen used the educational technology of OLAM to analyze the education and admission of colleges and universities and identified the reasons and potentially useful information affecting students' enrolment through data mining techniques based on the educational data of previous years [ 11 ].

To pursue the rationality of educational decision-making and promote educational decision-making from experience to science, scholars at home and abroad have done a lot of research and discussion on educational decision support from different perspectives. Du et al. started the theoretical research in this area by proposing a data mining-based decision support system for educational management [ 12 ]. Shen et al. used the B/S model and analyzed the process dynamics of learners in the Moodle learning platform using network analysis techniques. They saved the learner dynamic record data in the management system to form real-time analytical models or statistical reports [ 13 ]. Operations such as classification patterns, clustering patterns, association rules, and other analytical methods as well as visual representations were formerly used, and the latter were used to represent common model generation from behavioural dynamic data. Lee et al. used data mining techniques to estimate the potential of the subjects [ 14 ]. Based on the organic integration of information about learners' metacognition, motivational goals, knowledge acquisition, and learning attitudes, they established models with learners as the objects and predicted the future development tendencies of learners, analyzed and improved educational method models, explored the effectiveness of various applications to learning systems, and established data-based computing models to improve and enhance effective learning of learners [ 15 ].

We use literature, academic paper statistics, and content analysis methods to systematically sort out and summarize the various literature related to educational data mining that has been published in China and abroad and then conduct an overall study and analysis to objectively analyze and compare the current research status of data mining technology in the field of education at home and abroad [ 16 , 17 ]. The research trend of educational data mining in the future is discussed. Through the above literature, we can see that data mining is more oriented to the improvement of algorithms of data mining technology and the mining of existing education management data to analyze the intrinsic relationship between their different attribute values, but there is relatively little research on the valuable information that affects the relationship between new students' arrival rate and education management decisions. The purpose of this paper is to analyze and study educational management data from this perspective using mining techniques [ 11 , 18 ]. The development of an educational management information management system plays an important role in promoting the development of higher education informatization. Use data mining algorithms to establish models, test models, and application models to help educators understand the characteristics of education management as much as possible, analyze the factors that affect teaching quality, and maximize the value of data related to education management information as much as possible [ 19 ]. Give full play to the due role of educational data mining [ 20 ].

3. Research on the Decision Support System for Education Management Based on Data Mining

3.1. data mining education model construction.

Reasonable goals in the decision-making process are a prerequisite for rational decision-making. The formation of the decision goal, the size of the goal, and the decision maker's understanding of the goal will affect the smooth implementation of the decision [ 21 , 22 ]. In the process of establishing the goal, the nature, structure, and crux of the problem to be solved must be analyzed clearly before a reasonable decision target can be determined in a targeted manner. The decision goal must be very clear if the goal is too abstract or ambiguous, and ambiguous decision will be difficult to carry out, and the degree of achievement of the decision goal is also difficult to measure. The conceptual representation of the educational management decision support process is shown in Figure 1 . The education management decision-making process mainly includes four basic stages: target determination, design plan, implementation plan, and evaluation plan. Each stage of the educational management decision-making process has a close relationship with the decision-making environment.

Educational management decision support process.

After the decision objectives are determined, a variety of possible options are further designed for decision-makers based on feasibility studies. The proposed feasible options require a combination of overall exhaustiveness and mutual exclusivity to avoid bias in the option selection process. Overall exhaustiveness means that the various options developed should include all possible options found. Mutual exclusivity means that only one option can be chosen among different options [ 23 ]. These mutually exclusive options should all be easy for decision makers to compare and choose. The analysis, evaluation, and comparison of options mean that the indicators of all options are compared including the technical, economic, and social environmental conditions, factors, and potential problems, and dynamic financial indicators related to the decision objective are compared and evaluated. The possible constraints and potential problems of each option, as well as preventive and emergency measures, are also compared and evaluated. The proper execution of the user in the data mining process can accelerate the execution of the data mining process. Providing an interactive interface provides users with easy operation, and the interactive interface conveys the generated results to the user in a timely and convenient manner, and the generated results can be diverse.

The task of data integration is to merge data from multiple data sources into a unified data store for data mining. Data sources may include multiple databases, data cubes, and data files. The main considerations of data integration are attribute matching, data redundancy, and the detection and handling of value conflicts [ 24 ]. For attribute matching, attribute merging is performed by examining the field meaning of each attribute, data type, and so forth. Redundant data is considered at two levels. For attribute-to-attribute redundancy, the meaning of attribute entities is mainly examined, and attributes are streamlined by retaining fine-grained metrics of attribute values and ignoring coarse-grained metrics. This is because coarse-grained attribute data can come from the transformation of fine-grained data. Some redundant relationships between attributes are difficult to detect and can be detected by detecting how much an attribute can contain an attribute, such as by calculating the Pearson coefficient, also called the correlation coefficient of two attributes; for example, the correlation coefficient R ( X , Y ) given two attributes X and Y is shown in equation ( 1 ), where N is the number of tuples, x k and y k are the values of X and Y in tuple k , respectively, X ⟶ and Y ⟶ are the means of X and Y , and β ( X ) and β ( Y ) are the standard deviations of X and Y . The ID3 algorithm is used to establish a decision tree model based on the attributes of the database samples. The algorithm requires more logarithmic operations, which results in time complexity and low operating efficiency. The same formula needs to be calculated repeatedly, so the algorithm still has room for improvement. The algorithm needs to be optimized.

Facts are numerical measures, and multidimensional data models are organized around a fact, represented by a fact table that includes the name of the fact and the measure of the fact, as well as the code of each relevant dimension table [ 25 ]. A dimension is a pivot view or entity about which you want to keep records, and each dimension has a dimension table corresponding to it for data mining and decision analysis, which are Faculty Fact Table, Research Awards Fact Table, Research Results Fact Table, Research Funding Fact Table, Research Projects Fact Table, Talent Development Fact Table, Hardware Conditions Fact Table, and Subject Books Fact Table. Each fact table is associated with multiple dimension tables, such as discipline level, time, and unit level. The specific dimension tables are determined by the fact tables as shown in Table 1 .

Fact and dimension tables for data mining data warehouse.

After obtaining the discrete dataset, we first train to obtain a complete decision tree using the ID3 algorithm, which works as follows: let data D be partitioned into a training set of class-labeled tuples, assuming that the class-labeled attributes have N different values, and N different classes H i are defined ( i  = 1,…, N ). Let H iX be the set of tuples of class H i in X , and let | X | and | H iX | be the numbers of tuples in X and H iX , as in equation ( 2 ), where F ( X ) is the average expected amount of information needed to classify the tuples in X ; | X i | ∗F ( X i ) represents the weights of the i -th division; H i is the probability that any tuple in X belongs to class H i and is estimated by | X i |/| X |.

The Bayesian posterior theory is used to test each branch of the decision tree, and the branches that are not considered to have sufficient generalization ability or are unreliable are removed from the tree, resulting in a more compact tree [ 26 , 27 ]. The testing process is targeted at the knowledge of each rule translated from the decision tree, where each classification rule is obtained by searching from the root node of the decision tree top-down to a leaf node, and each classification rule consists of a conditional attribute tuple X and a conclusion classification label H x . For the test of each classification rule, we define two kinds of validation.

• Adequacy verification: G H i = ∑ i = 1 N H i x X . (3)
• Necessity validation: G H x | Y = H x , X X , G H x = H x X , (4)

We need knowledge with clear interpretability yet do not want the classification and prediction process to be too demanding, so to facilitate the simplification of the processing problem, we draw on fuzzy theory and perform some fuzzy optimization of the decision rules in classification and prediction. The affiliation function for a certain attribute value x belonging to a certain interval ( G 1 , G 2 ] is established as

A minimum acceptable value of affiliation u is specified, which can also be determined from a large number of experiments. In this paper, the value is set to u  = 0.9 to simplify the determination process.

The complexity of the educational system makes educational decision-making more difficult; an effective and rational educational decision support system will greatly contribute to the efficiency and rationality of educational decision-making. In this paper, we combine the characteristics of the educational decision-making process, the characteristics of the educational system itself, and the structural characteristics of the new generation decision support system to propose a decision support system model of educational management for the actual situation in the field of education and teaching.

3.2. Education Management Decision Support System Design Implementation

The data mining visualization system for education management information proposed in this study aims to provide decision support for education management information management in colleges and universities. The design goal of the system is to perform data mining on a large amount of education management data accumulated in the education management database over the years, and education management staff need to perform basic data analysis and data management on candidate data information through the online national college education management system. Finally, determine whether to admit or not and announce the admission results to the admitted candidates through the education management consulting system. The admitted candidates can also inquire about the situation of colleges and universities, previous years' education management admission score line, professional education management plan, application guidance, admission result, and issuance of admission notice through the education management information service system, while the relevant education management information managers of colleges and universities can also inquire about the historical information of education management and also use the data mining system to inquire about the application and admission and reporting situation through the system. Online analysis and visualization can be used to grasp education management information and make timely education management decisions.

The education management administration subsystem completes the function of maintaining information for education management records. All operations of the system require general records to be kept, and the general records are partially handled by the logs that participate in the shared section. The system starts when the user has successfully logged in and obtained the code that can control the scope of the system and expects to perform education management arrangement management operations. Education management arrangement includes 6 common functions of form presentation, adding, changing, deleting, finding, and detailing of education management teaching maintenance data. The administrator selects the hyperlink of education management and clicks on the specific list of each education management arrangement, and, with the help of clicking the function button given in the upper right corner, the system is commanded to complete the addition, deletion, and change of the list of activity plans.

A large part of the data application uses the report query mode for data service, and so does the education management analysis module. The flow of the report query is briefly explained below, and its flow chart is shown in Figure 2 . The first step of the report query is to log in to the system, at which time the system permissions are verified and only the functions and pages that the users are entitled to access are displayed; then the users enter the specific functional report interface according to their needs; the report interface functions are built based on various data analysis methods such as trend analysis, comparative analysis, cross-analysis, and retention analysis to establish the data model for query, and then the page is built based on the data model at the front end and the data organization at the back end. In the previous section, the detailed design and implementation of data storage and processing also mentioned that the data will be extracted from the dimension fields and indicator fields in the ADS layer, and now the query module can display the user-selectable dimensions based on the dimension field table. The user selects the dimension, the dimension value, and the data to send a query request, and the server calls the corresponding controller interface based on the requested URL and parameters to complete the query of the model data and return it to the Web client for display to the user.

Flow chart of report query.

The education management system designed and implemented in this paper uses a decision tree classification algorithm to generate a classification rule. One of the important tasks of the university's education management is to provide career guidance for students, so the education management system can use the information about students' schools to dig out their career direction. Therefore, the classification rules provided by the education management system for career guidance are as follows: Firstly, the students' school transcripts and comprehensive assessment are used as the input of the classification rules, and the courses taken by the students are classified as theoretical and practical courses; the spare time activities are classified as athletic and linguistic, and the students' roles in the activities are classified as leadership and nonleadership; based on this input information, the industry in which the students are employed is predicted. Finally, the employment information of previous students is used as an evaluation criterion. For example, grade analysis in university education management can obtain information about the knowledge points of students' grades in terms of question types and subject knowledge points of students' grades. It is very interesting to extract the important knowledge from the data warehouse of students' performance in higher education because there is a lot of information that can help to improve teaching management. The first step is to analyze the question types and the degree of correlation between question types, scores, knowledge points, and scores to summarize the strategies to train students.

4. Results and Analysis

4.1. data mining education model analysis.

To test the effect of the improved algorithm, the objective function can be scaled up and processed, and the number of misclassified entries can be calculated directly when calculating the error rate. The improved algorithm has an obvious effect on datasets with large data volume and complex data types, so the comparison of the convergence effect on the Vote dataset is listed, as shown in Figure 3 , for the convergence curves of the algorithm before and after the improved algorithm is carried out for 40 experiments and taken as the mean value. Through the experimental comparison, for the dataset with large data volume and complex data type, multiple experiments are conducted, and, within 500 iterations, the improved algorithm converges significantly faster than the unimproved one, is less likely to fall into the local optimal solution, and can converge to the global optimal value early.

Convergence curves of the number of misclassified rows in the Vote dataset.

Figure 4 shows the error rate of classification. When the proposed algorithm is used for classification, the error rates of all the datasets used are generally lower than those of NBC and NBC-W. If there is a dataset with a high classification error rate on the test data, it may be that the parameters set are not suitable, the training samples are not set reasonably well, or the data satisfy the assumption of conditional class independence. In general, NBC-IBA can classify accurately and reach the desired accuracy when dealing with most of the datasets. In terms of the running time of the algorithm, since the NBC-IBA algorithm needs to iteratively search for weights, the training time of the classifier is a little slower than that of NBC and NBC-W. After the training, not only is the speed comparable to that of NBC but also the accuracy rate is substantially improved in the subsequent classification process. In this chapter, the normalization criterion is applied when converting positions to weights, and the globally optimal positions are not simply used as weights.

Comparison of classification error rates of three algorithms for six datasets.

Figure 5 shows the comparison of the time run using the 3000 datasets collected in the object, and after the above analysis is applied to the actual one, from the perspective of the overall operation process, the advantages of optimization are very obvious, which not only reduces the number of calculation, but also calculates the order of all nodes, reducing the time complexity of operation. The optimized decision algorithm converts the original logarithmic operation into a simple four-rule operation during the calculation of the information gain, say application formula, which greatly reduces the average and overall computational effort. Therefore, the improved decision algorithm reduces the efficiency without changing the overall effect of the original algorithm and can completely replace the old algorithm for application to the system.

Running time.

4.2. Analysis of Educational Management Decision Support System

Figure 6 shows the comprehensive analysis, which shows the basic information of 2017–2020, the students' reports of different majors, different places of origin, and different subjects, and the final educational decision can be made by combining the educational data of four years as follows. The system analysis module shows that, except for economic management and foreign languages, which have reached the education standard in the last three years, all other key majors have not been able to reach the basic number of education standards, so it is necessary to adjust the education number of majors in this area. The specialties of computer science, environmental engineering, and other science and technology majors can be adjusted appropriately according to the actual situation to make the specialties more rational and to get more students enrolled and admitted in this way. From the above conclusions, we can see that many factors are affecting the enrolment rate, and then we can use this as a guiding basis in our future educational work to help the college decision-makers to plan the opening of courses and the adjustment of the setting of majors, as well as to give some scientific guidance to the work of discipline management departments as well.

Comprehensive analysis.

Figure 7 further verifies that there is a significant difference in overall assessment scores between the preceding teacher-training category and the conductive category and between male and female students, while it can be known that teacher-training category is higher than the conductive categpry, and female students are higher than male students. In terms of comprehensive assessment, The students' comprehensive assessment in the academic year is linearly related to their intelligence, and ability has a significant impact on comprehensive assessment. There is a significant difference between male and female students, with an average score of 69.96 for female students, which is higher than the average score of 60.10 for male students; there is a significant difference between teacher training and knowledge, with an average score of 76.33 for teacher-training students, which is significantly higher than the average score of 55.28 for knowledge students. There is a significant relationship between students' employment and their overall assessment and also with gender.

Correlation chart of comprehensive assessment scores.

The statistical analysis results of the whole school are used as an example of the display, as shown in Figure 8 , which shows the statistical interface of the bar graph of education management results. The following statistical charts can be saved online. Figure 8 takes the statistical graph of education management of each college of the whole university as an example and shows the specific distribution of 3 types of education management levels of each college according to the college as a division unit, from which it can be seen which college is worthy of attention, which college is worthy of praise, and so on; the majors of a certain college, classes of a certain major, students of a certain class, and so forth are applicable to assist decision-makers in making decisions.

Statistical chart of educational management results by faculty.

From the results of the above analysis, it is clear that there is a gap between male and female majors in educational technology in numerous aspects. To avoid this current gap phenomenon, the director of the program and the dean of instruction need to actively study and take favorable measures to guide male students to be competitive in academic life and other aspects to catch up with female students and narrow the gap. When setting the weighting coefficients for the comprehensive assessment of the student's academic year, the developers should focus on the coefficients for intellectual achievement and the criteria for ability bonus points. This is because these two parameters play a crucial role in the fairness and reasonableness of the comprehensive assessment of the student academic year. There is a need to continue to maintain the high weighting ratio of intellectual achievement in the comprehensive assessment. This will promote the all-around development of students' moral and intellectual and physical abilities while adhering to the principle that “students' primary responsibility is to learn” and giving full play to the effective means of reflecting students' learning in their intellectual performance. Professional teachers and administrators should gradually guide students to make efforts to study and stabilize their English grades from the time they enter the university, to achieve a relatively smooth passage of the fourth and sixth grades, avoiding the clinical learning style and the need to explore effective teaching methods to improve the English grades of male students. Although the results of comprehensive evaluation have an important impact on students' employment, students' single ability or special skills have a great advantage in employment. At the same time, in the employment process, the employment opportunities for men are higher than those for women, this through interviews and analysis of the employment situation in recent years, and the reason is mainly generated by the employment units in the treatment of gender differences between men and women and the point of conformity. Various activities are vigorously carried out to increase the opportunities for students to demonstrate their abilities and encourage them to take various certificate examinations to expand employment opportunities. Because of the current employment situation, female students should choose suitable majors according to gender characteristics when they are employed.

5. Conclusion

With the transformation of knowledge-based economy, the education industry is developing rapidly in this context, and modernization of education management is an important basis for promoting the sustainable development of the education industry, and an important symbol of modernization of education management is the construction of information system, which is an inevitable choice for the development of education modernization. In recent years, urbanization construction has been making breakthroughs, the population in need of education is relatively mobile, and the school layout has to be adjusted continuously as a result, which undoubtedly brings about serious problems to the education authorities, so it is extremely crucial to build an education management decision support system based on data mining. In this paper, the significance and background of the educational decision support management system are explained, and the user requirements, design scheme, and functional implementation are systematically described. The overall architecture and functions of the system are also analyzed, as well as the related system development environment and the required database design. The key technologies and difficulties involved in the development of the system are also discussed in detail. Finally, through testing the system, we found that the system can meet the user's requirements and can complete the basic education management work and has certain decision support functions. Of course, due to my limited ability, some functions of the system are not perfect, and the development of the system is still in the initial stage, so problems and shortcomings are inevitable, which are also the areas that need to be improved in the future. It is necessary to carry out research on the construction and application of various prediction models and monitoring models for the system prediction and dynamic monitoring mentioned in the discussion of the role of educational decision support system for evidence integration and benefit integration. These research skills will be a further extension and deepening of this research.

Acknowledgments

The work of this article was supported by Zhengzhou University.

Data Availability

Conflicts of interest.

The author declares that there are no conflicts of interest in this paper.

A decision support system for reducing the strategic risk in the schedule building process for network carrier airline operations

• Original Research
• Open access
• Published: 14 October 2022

Cite this article

You have full access to this open access article

• Muhammet Deveci   ORCID: orcid.org/0000-0002-3712-976X 1 , 2 ,
• Rosa Mª Rodríguez   ORCID: orcid.org/0000-0002-1736-8915 3 ,
• Álvaro Labella   ORCID: orcid.org/0000-0003-2764-1011 3 &
• Muharrem Enis Ciftci   ORCID: orcid.org/0000-0002-2067-2029 4 , 5  

2071 Accesses

5 Citations

Explore all metrics

This study addresses the evaluation of schedule time window of a new frequency for a network carrier airline. The ideal schedule for an airline can involve various criteria that consist of commercial and operational constraints. This study proposes a new integrated Best–Worst Method and Technique for Order Preference by Similarity to Ideal Solution based on heterogeneous decision making approach for determining the most suitable schedule. This approach combines the advantages of multi-expert multi-criteria decision analysis, which yields heterogeneous information, with a developed decision making model. In addition, a sensitivity analysis is performed to observe the robustness of the proposed approach. To illustrate the efficiency of the proposed approach, a real world problem at a network carrier airline in Turkey is presented. The results indicate that the flexibility and applicability of the proposed approach can address real-world problems.

Similar content being viewed by others

Aircraft Type Selection Problem: Application of Different MCDM Methods

A fuzzy multi-criteria approach for robust operating room schedules.

Sensitivity enriched multi-criterion decision making process for novel railway switches and crossings − a case study

Avoid common mistakes on your manuscript.

1 Introduction

Airline companies have many optimization and decision-making problems. They have been using Operations Research techniques to solve these problems since 1950 (Barnhart & Talluri, 1997 ). These problems include both long term, high cost and high uncertainties, such as fleet planning and short term, more certain, i.e., crew resource planning which is optimal crew planning level suitable for the next 1-month period. The fundamental airline problems can be categorized as planning or operational problems (Bazargan, 2016 ). The planning problems generally consist of flight scheduling (Çiftçi & Özkır, 2020 ), fleet assignment (Wei et al., 2020 ; Yan et al., 2020 ), aircraft routing (Chen et al., 2020 ; Cacchiani and Salazar-Gonzalez, 2020), and crew paring (Deveci & Demirel, 2018 ) and rostering (Quesnel et al., 2020 ). The operational problems comprise of revenue management (Yazdi et al., 2020 ), new destination study (Deveci et al., 2017 ), gate assignments (Xiao et al., 2020 ), and irregular operations.

The most valuable asset of an airline is its aircraft, and the most important product is its schedule. The revenue-generating activity of airlines is not the number of aircraft, destinations, or its in-flight design. What makes money for airlines is the seats they offer to the market. While inefficient schedule can cost millions of dollars, an efficient aircraft usage and schedule structure reduces the impact of fixed expenses (fixed and overhead costs) on production, resulting in increased gains. Many constraints and factors play critical role in schedule creation process. This process also varies according to the business model of the airline. The schedule planning of low cost airlines (LCC), and hub and spoke (HS) carriers are different from each other. While HS carriers optimize their schedule by considering connected flights and wide geographic areas and many destinations, LCCs consider individual routes and no connections are provided. HS carriers prefer high daily frequency, LCCs generally flies with lower frequency (Cook & Goodwin, 2008 ). In addition, the schedule structure that is suitable for the potential passengers of a city A, regardless of the airline, may not be suitable for another city B (Belobaba et al., 2015 ).

In airlines, schedule planning is generally done by network or schedule planner. Network planning can be defined as the managing passenger flow and flight connections at the hub and spoke system. Network planners are responsible for carrying out the economic evaluation of various new route options while independently assessing various business risks such as demand planning, competitive landscape, cost implications and financial exposures. They also consistently review route network performance in order to identify profitability issues and provide forewarning to the senior management and executive teams along with detailed recommendations to improve route profitability and strategy going forward. Network planning includes evaluation of the strategic opportunities in the product planning, such as aircraft redeployment scenarios and new schedule design by providing improvements in network (fleet, schedule and routing) deployment to maximize network profitability (cost efficiencies and revenue potential). A typical network planner starts analyzing the market data and identifying patterns for potential routes in order to optimize airline schedule and network profitability (Bazargan, 2016 ).

Creating a new schedule is a complex process that requires the contribution of internal and external stakeholders from different departments. Examples of internal stakeholders are operational departments, crew planning, revenue management etc., and external stakeholders are passengers, civil aviation authorities, airport slot planning departments, ground handling firms etc. Experts always evaluate more than one alternative schedule for the new frequencies and destinations. Therefore, a Multi-Expert Multi-Criteria Decision Analysis (MEMCDA) process is necessary to perform this complex selection process. It should consider different points of view from internal and external stakeholders and multiple conflicting criteria that might be quantitative or qualitative. This implies a heterogeneous decision framework (Herrera et al., 2005 ; Palomares et al., 2013 ) able to deal with different kind of information and the imprecision of the experts involved in the problem.

In order to select the new frequency for an airline company different Multi-Criteria Decision Making (MCDM) models introduced in the literature (Chen & Hwang, 1992 ; Kahraman et al., 2015 ) such as, AHP, TOPSIS, VIKOR etc., could be used to solve this decision problem. Nevertheless, due to the interdisciplinary of this decision process that implies the necessity of using a heterogeneous framework, as far as we know there is not any previous model that can straight to solve this MCDM problem.

The purpose of this study is to propose a new and specific MEMCDA approach able to deal with different kinds of information and to decide on the ideal schedule structure for a new frequency by using commercial and operational constraints within existing network. Among the different MCDM models that can be applied to solve this problem, in this contribution we will use fuzzy TOPSIS because it is one of the most widely used models in MCDM to solve different problems obtaining satisfactory results (Behzadian et al., 2012 ; Sang et al., 2015 ) because of its advantages regarding other MCDM models (Ishizaka & Nemery, 2013 ; Shih et al., 2007 ): (i) it has a sound logic that represents the rationale of human choice, (ii) a scalar value that considers the best and worst alternatives at the same time, (iii) a simple computation algorithm and a (iv) minimal number of inputs from experts.

On the other hand, as the criteria considered to select the new frequency have different importance, we will use the Best Worst Method (BWM) (Rezaei, 2015 ) to obtain the criteria weights because it reduces the inconsistency in experts’ preferences in comparison with other methods as AHP (Kahraman et al., 2015 ). Moreover, the dependencies among criteria are studied and taken into account by using the Trapezoidal Fuzzy Number Weighted Extended Bonferroni Mean (Dutta et al., 2019 ) which reflects the criteria importance.

Therefore, the main novelties of the proposal are the following ones:

To define a new MEMCDA to model different kinds of information and able to provide the best schedule for a new frequency according to commercial and operational constraints.

To use the BWM to obtain the criteria weights by means of the experts’ opinions.

To study the relations among criteria and use a suitable aggregation operator able to capture such relations and consider the criteria weights.

To evaluate a new frequency for a network carrier airline between Istanbul and Stockholm applying the proposed MEMCDA model.

To show the robustness of the solution by a sensitivity analysis.

The rest of paper is organized as follows: Sect.  2 introduces the factors taken into account to choose a schedule and the operational constraints. It makes also a short review about fuzzy MCDM methods related to transport problems. Section  3 proposes the MEMCDA able to deal with heterogeneous information to select a new frequency for a network carrier airline. Section  4 presents a real case study to show the performance and feasibility of the proposal. It also includes a sensitivity analysis to study the robustness of the solution obtained, and finally Sect.  5 points out some conclusions.

2 Background

This section revises the general factors considered to choose a schedule for a new frequency in a network carrier airline, explains the operational and commercial constraints and shows different fuzzy MCDM approaches that have been used to manage air transport problems.

2.1 General factors for schedule planning

There are models that evaluate the schedules and forecast its market shares while developing and calibrating models that quantifies the passenger choice on the picking up the airline for their itinerary. These models are generally called quality service index (QSI) models. It is a method to evaluate different options (airlines and flights) in front of the consumer (passenger). It starts determining the factors that affect passengers’ choice when choosing a flight among the others. Passenger utility is a value that is calculated by these models and it assumes that is going to be maximised with rationale choices of experts. General factors that have been used to the selection are the following ones (Belobaba et al., 2015 ):

2.1.1 Number of stops

How many stops/connections occur in the itinerary? Some lucky city pairs in the world have direct flights (IST-JFK, LAX-DXB etc.), however given 10 K airports in the world, there are many more indirectly connected city pairs (ADB-BCN, ESB-LHR etc.) with at least 1 or more stops. Passenger utility decreases while providing an increase of the number of stops in the travel.

2.1.2 Aircraft type

Which aircraft type is going to operate the flights? This factor is important especially for the jet and turboprop aircraft types. There is a less preference on the turboprops over the jet aircrafts. In most cases, passengers are not aware of different aircraft types involved in a given itinerary. However, with help of the advertisement, airlines can make more revenues; becoming the first airline to operate the newest aircraft type (e.g. Airbus 380, Boeing 787) or the being the airline with the youngest fleet.

Aircraft type is also important in terms of its capacity (available seat) provided to route. High capacity attracts more market share.

2.1.3 Flight frequency

Just like aircraft type, more frequent services provide more capacity and attract more market share. It is also important to have at least daily services (one flight per each day in a week) in order to cover all the demand around the week.

2.1.4 Detour

Comparison (ratio) of the direct routing and indirect routing in terms of distance. Nonstop itineraries detour is 1. In general, detour factor up to 1.4 is acceptable for the itineraries that have intermediate stops. Although, passengers do not prefer high detours, they are obliged in some cases due to insufficient itinerary options. There can be only one flight to some airports and they do not have any option to select (Burghouwt & Wit, 2005 ).

2.1.5 Travel time

Elapse time or travel time can be defined as total trip time that is required from origin city to destination city of itinerary including connection times at the intermediate stops. Longer itineraries are less attractive compared to shorter ones.

2.1.6 Time of day preference

Morning and evening times are important for business travellers. It is also important to match hotel check-in and check-out timings for leisure travellers. Night schedules, especially after the midnight, are less preferable due to less transportation opportunities between airports and city centres, inconvenient departure and arrival times of the flight. Destinations with high local share are scheduled according to their time-of-day preferences in order to ensure market acceptance and exploit market potential.

2.1.7 Day of week preference

Mondays and Fridays are important for business travellers in general. It is important to have schedule on weekends due to high demand for leisure travellers. In some Muslim countries in the Middle East, Friday and Saturday are weekends, therefore airlines should be careful and aware of this fact while they plan their schedule to these destinations.

A typical schedule consists of the following information (see Table 1 ); airline code, flight number, departure time, arrival time, aircraft type, block time and departure day.

2.2 Operational and commercial constraints

Flight scheduling has a strong impact on all of the activities of the airlines (Bazargan, 2016 ). Operations, revenue management, crew planning, profitability are affected by the schedule structure. Building a perfect schedule is constrained by both economic and operational constraints. A schedule is successful when it is commercial profitable as well as operationally feasible. Thus, it is convenient to consider some operational and commercial constraints.

Operational constraints:

Block times of the flight legs and Ground times at the stations should be validated by the operational departments.

Departure and arrival times of the schedule should be in line with the meteorological analysis. Destinations that have airport curfews and do not allow night operations should be planned according to their respective airport curfews.

Minimum connection time between flights that are required for transferring passengers at the hub station must be considered when the schedule is planned.

Departure and arrival slots at the congested spoke airports must be satisfied.

Crew planning department should validate the duty times of the schedule.

Commercial constraints:

Fleet should be available and should be rotated for given schedule. Local departure and arrival times should be reasonable for passengers. Market potentials are used to identify the ideal capacity allocation for the destinations. Historical market data, growth rates and the level of competition are used to determine the market potential for each destination. Together with passenger demand, potential of the belly cargo contribution of the destinations should be considered when feasibility studies are evaluated.

Fleet assignment should be in line with the market potential and passenger preference. Passenger spill is minimized by re-distributing aircraft capacity in order to capture full potential of passengers. Seat capacity for seasonal destinations should be adjusted through the year in order to reflect demand variability.

A minimum service level of frequency per week should be defined in order to guarantee product quality. Frequency rights should be utilized accordingly, however frequency or capacity cannot exceed the defined right in the bilateral air service agreement to allow international commercial air transport services between countries.

Schedule of the codeshare partner airlines should be considered for codeshare connecting passengers.

2.3 Fuzzy MCDM approaches in air transport management

There have been studies investigating different methods for various air transport management problems over the last decade as presented in Table 2 . The acronyms are defined as follow: AHP is Analytic Hierarchy Process, ANP is Analytic Network Process, TOPSIS is Technique for Order Preference by Similarity To An Ideal Solution, VIKOR is VIseKriterijumska Optimizacija I Kompromisno Resenje, DEMATEL is DEcision MAking Trial and Evaluation Laboratory, GRA is Grey Relational Analysis, QFD is Quality Function Deployment, WASPAS is Weighted Aggregated Sum Product Assessment, ARAS is Additive Ratio ASsessment, and COPRAS is COmplex PRoportional Assessment.

A variety of fuzzy MCDM approaches have been applied to air transport management problems by using different fuzzy extensions to model the uncertainty and vagueness of the information. For instance, Tsaur et al. ( 2002 ) proposed an approach based on AHP and TOPSIS to evaluate the service quality of airline using fuzzy sets, Kuo ( 2011 ) used interval-valued fuzzy sets based VIKOR and GRA, Percin (2018) introduced another fuzzy approach based on DEMATEL, ANP and VIKOR, and Deveci et al. ( 2018 ) used interval type-2 hesitant fuzzy sets to model the uncertainty and defined a new MCDM model.

Some researchers defined new fuzzy MCDM approaches to evaluate the quality of airports: fuzzy sets based TOPSIS (Wang & Lee, 2007 ), and VIKOR and GRA (Kuo & Liang, 2011 ). Liou et al. (2011), and Garg (2016) presented a novel approach based on ANP, and AHP based TOPSIS, respectively, dealing with fuzzy sets for strategic alliance partner selection problems. Torlak et al. ( 2011 ) applied fuzzy TOPSIS approach to rank air carriers according to business competition. Deveci et al. ( 2017 ) studied airline new route selection between Turkey- North American region destinations using interval type-2 fuzzy sets based TOPSIS.

3 A heterogeneous decision making approach

This section proposes a selection process based on a fuzzy TOPSIS method that provides a rank of frequencies to include a new one in the schedule planning for airlines. It will be able to deal with heterogeneous contexts in which linguistic and numerical values are used to evaluate the criteria. Additionally, the criteria weights are obtained by means of the BWM. This selection process consists of six phases (see Fig.  1 ) which are explained in further detail below.

Scheme of the heterogeneous decision making approach

3.1 Definition of the framework

A set of experts \(E=\{{e}_{1},\dots ,{e}_{m}\}\) provides their preferences over a set of alternatives \(X=\{{x}_{1},\dots ,{x}_{n}\}\) that are defined by a set of main criteria \(C=\{{c}_{1},\dots ,{c}_{r}\}\) where each main criterion is defined by a set of sub-criteria \({c}_{i}=\left\{{c}_{i1},\dots {c}_{it}\right\}, {c}_{i}\in C\) .

The experts’ preferences \({e}_{k}\in E\) over the alternatives \({x}_{l}\in X\) and sub-criteria \({c}_{ij}\in C\) are represented by preferences vectors: \(({p}_{ij}^{kl},\dots ,{p}_{rt}^{kl})\) with \(i\in \left\{1,\dots ,r\right\}\) and \(j\in \left\{1,\dots ,t\right\}\) . In this proposal the preferences \({p}_{ij}^{kl}\) can be elicited by means of different expression domains (linguistic terms and numerical values) according to their nature. Therefore,

The main criteria and sub-criteria weights are obtained from experts’ opinions by using the BWM and they are represented by vectors: \(({w}_{1}^{k},\dots ,{w}_{r}^{k})\) and \(\left({w}_{i1}^{k},\dots ,{w}_{it}^{k}\right)\) .

3.2 Gathering of information

Once the framework has been defined, experts \({e}_{k}\in E\) involved in the selection process elicit their preferences about the alternatives \({x}_{l}\in X\) and sub-criteria \({c}_{ij}\in C\) by using linguistic terms or numerical values according to the criteria nature and provide their opinions about the criteria importance by using a scale of values that is used by the BWM to obtain the criteria weights. This method is explained in the following phase.

3.3 Applying BWM to obtain the criteria weights

The criteria and sub-criteria weights are computed through the BWM (Labella et al., 2021 ; Rezaei, 2015 ). The BWM is a MCDM technique aims to derive the prioritization of different decision elements by means of pairwise comparisons. The method consists of comparing the best and worst element with the remainder, as opposed to other proposals where all elements are compared with each other. These comparisons are so-called reference comparisons in BWM. In this way, the number of reference comparisons is reduced and, in turn, the emergence of inconsistency in experts’ preferences that appears when the number of comparisons is too large. The BWM steps are described below:

To choose a set of decision criteria. In our proposal, such criteria are described in Sect.  4 .

To select the best criterion \({C}_{B}\) and the worst criterion \({C}_{W}\) . If there are several best and/or worst criteria, they are selected randomly.

To make pairwise comparisons among \({C}_{B}\) and the rest of the criteria, by obtaining the Best to Others (BO) vector, \(BO=\left\{{a}_{B1},{a}_{B2},\dots ,{a}_{Br}\right\}\) , where \({a}_{Bi}\) represents the preference degree of \({C}_{B}\) over the criterion \({C}_{i}\) and \({a}_{B1}\ge 1, i=1, 2,\dots ,r, i\ne B\) .

To make pairwise comparisons among the rest of the criteria and \({C}_{W}\) , by obtaining the Others to Worst (OW) vector, \(OW=\left\{{a}_{1W},{a}_{2W},\dots ,{a}_{rW}\right\}\) , where \({a}_{iW}\) represents the preference degree of the criterion \({C}_{i}\) over \({C}_{W}\) and \({a}_{iW}\ge 1, i=1, 2,\dots ,r, i\ne B or W\) .

To compute the criteria weights by using an optimization model. For each reference comparison, the optimal criteria weights must satisfy \({w}_{B}/{w}_{i}={a}_{Bi}\) and \({w}_{i}/{w}_{W}={a}_{iW}\) . Hence, the maximum absolute differences \({|w}_{B}/{w}_{i}-{a}_{Bi}|\) and \({|w}_{i}/{w}_{W}-{a}_{iW}|\) should be minimized (see (M-1)).

where ε refers to the maximum absolute deviation between the reference comparisons provided by the experts and the computed criteria weights \(\left({w}_{1},{w}_{2},\dots ,{w}_{r}\right)\) by the model (M-1).

A key aspect in the BWM is related to the experts’ preferences consistency. Obviously, experts’ preferences should make sense and not be provided in an illogical or random way. For this reason, in (Rezaei, 2015 ) was introduced a consistency ratio to measure the level of inconsistency in experts’ opinions. According to Rezaei, perfect consistency is achieved when \({a}_{Bi} x {a}_{iW}={a}_{BW}\) . From this assumption, the consistency ratio is computed as follows:

where \({\varepsilon }^{*}\) represents the maximum absolute difference between the optimal weights obtained from the model (M-1) and the reference comparisons provided by the experts. Consistency index is a numerical vale obtained from \({a}_{BW}\) and several experiments carried out by Rezaei (see Rezaei ( 2015 ) for further detail). The consistency ratio provides a value in [0, 1], where 0 represents perfect consistency.

3.4 Unification process

The heterogeneous information provided by experts must be transformed into a common expression domain to facilitate the computations. We use a fuzzy domain to model the uncertainty and carry out the computations in a precise way. This unification process is reached by means of different equations according to the type of information.

Linguistic terms : The linguistic terms \(S=\{{s}_{0},\dots ,{s}_{g}\}\) are transformed into trapezoidal fuzzy numbers which are represented as \(\widetilde{Z}=(a,b,c,d)\) .

Numerical values: The numerical values are normalized into \([\mathrm{0,1}]\) and then transformed into trapezoidal fuzzy numbers by the following function \(F\) .

For sake of clarity the experts’ preferences \({{\varvec{p}}}_{{\varvec{i}}{\varvec{j}}}^{{\varvec{k}}{\varvec{l}}}\) transformed into trapezoidal fuzzy numbers are represented as \({\widetilde{{\varvec{p}}}}_{{\varvec{i}}{\varvec{j}}}^{{\varvec{k}}{\varvec{l}}}\) .

3.5 Aggregation process

Once the criteria weights are computed by the BWM, they are used to obtain the overall values for the main criteria and alternatives. This process is divided into two phases:

Criteria aggregation: Experts’ preferences \({\widetilde{p}}_{ij}^{kl}\) over the sub-criteria \({c}_{ij}\in C\) for each alternative \({x}_{l}\in X\) are fused by means of a fuzzy aggregation operator to obtain an overall value \({\widetilde{p}}_{i}^{kl}\) . We suggest using the Trapezoidal Fuzzy Number Weighted Extended Bonferroni Mean (TFNWEBM) (Dutta et al., 2019 ) because it allows capturing heterogeneous relations among the input (in this paper sub-criteria) and reflects the criteria importance. This operator classifies the input into two categories \(U\) and \(V\) , where every input of \(U\) is related to the remaining inputs, i.e., \({E}_{i}\subset a\backslash \{{a}_{i}\}\) and the inputs of \(V\) are not related among them.

Definition 1

(Dutta et al., 2019 ) : Let \(\left({\widetilde{Z}}_{1},\dots ,{\widetilde{Z}}_{n}\right)\) be a vector of trapezoidal fuzzy numbers, which are interrelated. For any \(p,q\ge 0\) with \(p+q>0\) and the weighting vector \(({w}_{1},\dots ,{w}_{n})\) such that \({w}_{i}>0\) and \(\sum_{i=1}^{n}{w}_{i}=1\) , the aggregated value by the TFNWEBM is a fuzzy number and it is given as follows:

\(\widetilde{Z}=(a,b,c,d)\) is a trapezoidal fuzzy number for all \(i=1,\dots ,n\) . The \({WEBM:[\mathrm{0,1}]}^{n}\to [\mathrm{0,1}]\)

where \({I}_{i}, {E}_{i}\) is the set of indices of the elements of , \(I{^{\prime}}\) is the set of indices of the inputs of \(V\) and \(|{I}{^{\prime}}|\) is the cardinality of the set \(I{^{\prime}}\) . The empty sum of fuzzy numbers is set as fuzzy zero (with the representation (0,0,0,0) following the classic convention for crisp system (for further details see Dutta et al. ( 2019 )

Experts aggregation: The overall values \({\widetilde{p}}_{i}^{kl}\) obtained in the previous step are fused by using the fuzzy weighted aggregation operator to obtain a global value for each main criteria and alternative \({\widetilde{p}}_{i}^{l}\) . We propose this aggregation operator because it allows to assign different weights to the experts involved in the MCDM problem according to this knowledge or experience.

where \({w}_{k}\) is the weight assigned to the expert \({e}_{k}\) , \({w}_{k}>0\) and \(\sum_{k=1}^{m}{w}_{k}=1\) where is the weighted form of Extended Bonferroni Mean aggregation operator given by))

3.6 Applying Fuzzy TOPSIS

Finally, in order to obtain a ranking of frequencies and select the best one for the airline, the fuzzy TOPSIS method (Chen & Hwang, 1992 ) is used. It is explained in short as follows:

To create the fuzzy normalized decision matrix \(\widetilde{D}={({\widetilde{p}}_{i}^{l})}_{nxr}\) by means of the global values obtained in the previous phase.

To compute the weighted fuzzy normalized decision matrix \(\widetilde{R}={({\widetilde{v}}_{i}^{l})}_{nxr}\) being \({\widetilde{v}}_{i}^{l}={\widetilde{p}}_{i}^{l}*{w}_{i}\) , with \({w}_{i}\) the main criteria weight and \({w}_{i}>0\) , \(\sum_{i=1}^{r}{w}_{i}\) = 1.

To define the positive ideal solution (PIS) \({\widetilde{Z}}^{+}=\left({\widetilde{z}}_{1}^{+},\dots ,{\widetilde{z}}_{r}^{+}\right)\) , and the negative ideal solution (NIS) \({\widetilde{Z}}^{-}=\left({\widetilde{z}}_{1}^{-},\dots ,{\widetilde{z}}_{r}^{-}\right)\) , being \({\widetilde{z}}_{i}^{+}=\left(\mathrm{1,1},\mathrm{1,1}\right)\) and \({\widetilde{z}}_{i}^{-}=\left(\mathrm{0,0},\mathrm{0,0}\right)\) .

To compute the distance for each alternative from \({\widetilde{Z}}^{+}\) to \({\widetilde{Z}}^{-}\) .

where \(d(\bullet ,\bullet )\) is the distance between two trapezoidal fuzzy numbers and \(l=\{1,\dots ,n\}\) .

To compute the closeness coefficient \({{\varvec{C}}{\varvec{C}}}^{{\varvec{l}}}\) for each alternative:

Finally, the alternatives (new frequencies) are ordered according to \({{\varvec{C}}{\varvec{C}}}^{{\varvec{l}}}\) to select the best one.

4 Case study

This section describes a real case study to include a new frequency in the route Istanbul and Stockholm that is solved by using the proposed heterogeneous decision making approach. Moreover, a sensitive analysis is introduced to show the robustness of the decision.

4.1 Selection of new frequency

Four alternatives are identified for selecting the most appropriate new frequency. Table 3 presents the current flight schedule (in local times) of airline that operates between Istanbul and Stockholm on daily three basis. The proposed 4 new frequency alternatives are given in Table 4 . The visualization of current and new frequencies is shown in Fig.  2 . While current schedule departure and arrival times are shown in red colour, new schedules are represented in blue colour. 3 black horizontal axes represent the 24 clock hour in one day for Istanbul and Stockholm. If a flight is below the horizontal axis, it means that it is an arrival flight. If a flight is above the horizontal axis it means that it is a departure flight.

Current and alternative schedules on timeline

The new frequency alternatives are described by twelve evaluation sub-criteria under four main criteria including passenger preference, competition, slot availibiliy and connection. These main criteria and sub-criteria have been determined and defined by airline company experts. Figure  3 presents a schematic overview of the qualitative and quantitative criterias that are used in the study.

Scheme of the main criteria, sub-criteria and their kinds of information for prioritizing new frequencies

The main criteria and sub-criteria are defined as follows:

Passenger schedule preference (C 1 ): It is defined as the time preference of the passenger for an alternative schedule. The departure time and days preference are examined in Fig.  4 a, b for criterion C 11 and C 13 . Both figures present a preference coefficient on the vertical axis which creates a curve over the day of a sample week. These values are calculated from historical market data with the help of experts using statistic tools.

C 11 : Hub local departure time: Scheduled time of departure of a flight from hub which shows the doors closing time at the gate.

C 12 : Dest local arrival time: Scheduled time of arrival of a flight to spoke which shows the doors opening time at the gate. C 13 : Dest local departure time: Scheduled time of departure of a flight from spoke airport which shows the doors closing time at the gate. C 14 : Hub local arrival time: Scheduled time of arrival of a flight to hub which shows the doors opening time at the gate.

Departure time and day preference for criterion C 11 and C 13

Competition (C 2 ): Competition criterion reflects the effects of the other schedules on the same city pair. Other schedule could belong a competitor airline or it could be the current schedule of the examined airline Schedule (see Table 5 ).

C 21 : Schedule time of competitor airlines: The aim of this criterion is to present the competitiveness of the alternative schedule by comparing with other airlines' schedule that serves the same city pair (IST-ARN-IST route).

C 22 : Cannibalization effect current schedule: The aim of this criterion is to show the deterioration of the alternative schedules by comparing with current schedule of the case airline on the IST-ARN route. ie, Case airline has 3 flights on IST ARN route. Alternative schedule will have effect on the local and transfer passenger demand on the other flights of the examined airline. Alternative flights will have its own demand and it will also steal market share from competitors and current schedule of the airline. Fig. 5 shows the seat load factor loss of the alternative schedules on the current schedule of the examined airline. These values are calculated by schedule experts using airline planning simulation tools.

Cannibalization impact of the alternative schedules on the current flights

Slot availability (C 3 ): Slot is defined as a landing and departing permission from airport authority to use the airport, runway and terminal for a specific time range. Slot availibility is shown in Fig.  6 . which illustrates the probability of getting a slot for alternative schedules. These values are provided by airport authorities to the airline schedule planners.

C 31 : Hub slot availability: Probability of having a slot at the hub airport (IST) at planned schedule time.

Slot availability of alternative schedules

C 32 : Dest slot availability: Probability of having a slot at the spoke airport (ARN) at planned schedule time.

Connection (C 4 ): Connection criterion is a measure for connectivity of the alternative schedule at the IST airport and ARN airport within a time window. This time window starts with minimum connection time (1 h) until 12 h for a connection. High number of connections creates more demand for the planned schedule, therefore it is very important for a profitable schedule. Figure  7 presents the number of connections for alternative schedule on bar charts as primary axis on the left. Black curve represents the passenger volume on the connections which is secondary axis on the right. Number of connection values are calculated on the schedule by simply counting the flight legs that have sufficient connection time. Passenger volumes are the market figures that are flown in the last one year.

Number of connection opportunities and passenger potentials of the alternative schedules

C 41 : # of weekly ınbound connection: An Inbound connection is that flights are arriving to hub and feeding the connection of the specified destination in terms of passenger volume.

C 42 : # of weekly outbound connection: An Outbound connection is that flights are departing from hub and defeeding the connection of the specified arrival in terms of passenger volume.

C 43 : # of weekly codeshare connectivity: A codeshare connection is a connection that has at least a carrier change and a flight change in the itinerary with a codeshare partner airline.

C 44 : dep + arr connection potential: Total number of O&D market volume in terms of passenger.

It is necessary to consider that in this case study experts involved in the problem point out that some sub-criteria are related (see Fig.  8 ). The sub-criterion \({c}_{12}\) is dependent of \({c}_{11}\) and sub-criterion \({c}_{14}\) is dependent of \({c}_{13}\) .

Dependency among sub-criteria

4.2 Applying the novel heterogeneous decision making approach

The case study introduced in the previous section is solved by using the MEMCDA approach presented in Sect.  3 . To facilitate the understanding of the case study resolution, the different steps of the proposal applied to it are described in detail in the following subsections. Note that the resolution of the case study has been carried out by using the decision support system FLINTSTONES (Estrella et al., 2014 ).

4.2.1 Definition of the framework

Six experts from network planning and scheduling department of an airline company evaluate the four possible new frequencies between Istanbul and Stockholm over 4 criteria and 12 sub-criteria. These experts are specialized in network planning and scheduling and each of them have at least 5 years’ experience. Both alternatives, criteria and sub-criteria have been described in Sect.  4 .

4.2.2 Gathering of information

This case study is composed by quantitative and qualitative criteria. The experts do not need to provide their opinions over the quantitative criteria, since they represent objective information related to different aspects of the new frequencies (see Table 6 ) as they have been explained for each criterion. However, the experts should provide their opinions over the criterion Competition and its sub-criteria related to the effects of the other schedules on the same city pair. To evaluate such criterion and sub-criteria, the experts provide qualitative assessments by making use of the following linguistic terms set S  =  {Nothing (N), Very low (VL), Low (L), Medium (M), High (H), Very high (VH), Excellent (E)} . The qualitative preferences are shown in Table 7 . Additionally, the experts provide their opinions over the criteria importance by means of pairwise comparisons, which will be used to derive the weights by using the BWM. Such pairwise comparisons are presented in Tables 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 in Appendix. Note that these opinions were obtained from a questionnaire which was sent to the experts via email. The structure of this questionnaire follows the BWM approach thus, after a brief description of the problem and the criteria, the experts were asked to choose the best and worst criteria according to their expertise to lately compare these with the remainder. They had to make this selection for both the main criteria and the sub-criteria which belong to each one. An example of this questionnaire can be found at the following link https://sinbad2.ujaen.es/sites/default/files/2022-07/Survey__Experts.pdf .

4.2.3 Applying BWM to obtain the criteria weights

From the pairwise comparisons given in Tables 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , the criteria and sub-criteria weights are derived by using the BWM, particularly the optimization model (M-1). Table 8 presents the resulting weights for all the criteria and sub-criteria together the consistency of the experts’ opinions. Note that, according to (Liang et al., 2020 ), all the experts’ preferences are consistent.

4.2.4 Unification process

The heterogeneous information about the new frequencies implies the need of transforming such information into a unique expression domain in order to accomplish the computations in the next step. The unification process defined in Eq. ( 1 ) provides a fuzzy representation of each expert’s assessment for NA 1 (see Table 9 ). The calculations for the rest of the alternatives (NA 2 , NA 3 , and NA 4 ) are presented in Tables 25 , 26 , 27 in Appendix.

4.2.5 Aggregation process

First, for each expert, the sub-criteria are aggregated by means of the TFNWEBM operator in order to obtain an aggregated value for each criterion. In this step, the criteria and sub-criteria weights derived from the BWM are used. The experts’ decision matrices obtained from this aggregation step are shown in Table 10 .

Afterwards, each expert’s decision matrix is aggregated by using the Weighted Average Mean operator in order to obtain a collective decision matrix (see Table 11 ). In this step, the experts’ weights are derived from the years of experience of each expert (see Table 12 ).

4.2.6 Applying Fuzzy TOPSIS

Finally, the ranking of the alternatives is obtained by applying the fuzzy TOPSIS approach revised in Sect.  3.6 (see Table 13 ).

According to the results obtained from our proposal, the best frequency is NA 3 .

Therefore, we have proved that the proposal is useful to solve real-world MEMCDA problems such as the one presented in the case study. The unification process allows transforming the heterogeneous information provided by the experts into a single format that facilitates computations. Then, the BWM derives the weights for criteria and sub-criteria by using an optimization model that guarantees the most representative weights according to the experts’ opinions. Afterwards, the aggregation process takes into account the relations between criteria and their importance by using the TFNWEBM operator and the different importance for the experts according to their level of expertise by using the Weighted Average Mean. Finally, taking advantage of the fuzzy representation, the fuzzy TOPSIS method is applied to obtain the ranking of the alternatives.

4.3 Sensitivity analysis

The robustness of the given solution is analyzed by means of a sensitive analysis (Triantaphyllou, 2000 ). This sensitive analysis consists of identifying the most critical criterion, which is the one that, with the smallest change in its weight, implies a change in the ranking of the alternatives. Table 14 shows the necessary changes in the criteria weights to modify the position between each pair of alternatives, which are graphically represented in Fig.  9 . According to the results, the most critical criterion is C 2 since, an increment of the 45.04% (highlighted bold in the table) on its weight would provoke an exchange of positions between the alternatives NA 3 and NA 4 . The remainder criteria need very high changes on their weights to provoke the same situation and, in some cases, the exchange of positions between specific pair of alternatives never happens (represented as Non Feasible (N/F)). Therefore, the results presented in Table 14 indicate that our solution is completely robust being necessary to modify the weights more than 45% to change the ranking of the alternatives.

Sensitive analysis for each criterion

5 Conclusions

This study aims to propose a new and specific MEMCDA model based on heterogeneous information for solving the selection problem of a new frequency for a network carrier airline in Turkey. The main advantages of this proposal are:

It is able to deal with different kind of information to evaluate the criteria

It provides the best schedule for a new frequency according to commercial and operational constraints.

It uses the BWM to obtain the criteria weights by means of the experts’ opinions.

It studies the relations among criteria and use a suitable aggregation operator able to capture such relations and consider the criteria weights.

Our study helps airline network and schedule planners to manage the potential risks through at the strategic planning phases of the schedule building process. Thanks to considering the slot availability criteria, experts considers the runway and gate congestion at the airports and they are able to enhance the robustness and resilience of the airline schedules at the operational phase.

The limitations of this study are as follows: (i) profitability evaluation of the alternative schedule is not possible due to confidentiality issues. Therefore, we cannot conclude that the best alternative is also the best profitable one. There might be revenue differences on the different time of days due to the different mix of passengers and its volume. (ii) it has been pre-assumed that alternative schedules are operationally feasible in terms of meteorological conditions and airport operations. In case of the non-compliance with these constraints, schedules cannot be operable. Additionally, regarding the MEMCDA approach, (iii) the results are represented both with a numerical and fuzzy representation but a linguistic representation closer to the experts’ way of thinking may facilitate even more their readability from the experts’ point of view.

As future works, the proposal of a new MEMCDA approach able to obtain easy-to interpret linguistic results may be interesting. Additionally, a consensus reaching process may be included in the resolution scheme of the MEMCDA approach with the aim of detecting and smoothing possible disagreements in the experts’ preferences and obtain agreed solutions. At the same time, the model can be made more effective by using operation research techniques. We can study the application of dynamic methods that allow the experts in the process consider the evolution across time to select a new frequency for a network carrier airline.

Barak, S., & Dahooei, J. H. (2018). A novel hybrid fuzzy DEA-Fuzzy MADM method for airlines safety evaluation. Journal of Air Transport Management, 73 , 134–149.

Article   Google Scholar  

Barnhart, C., & Talluri, K. T. (1997). Airline operations research. Design and operation of civil and environmental engineering systems, 435–469.

Bazargan, M. (2016). Airline operations and scheduling . Routledge.

Book   Google Scholar  

Behzadian, M., Otaghsara, S. K., Yazdani, M., & Ignatius, J. (2012). A state-of the-art survey of TOPSIS applications. Expert Systems with Applications, 39 (17), 13051–13069.

Belobaba, P., Odoni, A., & Barnhart, C. (Eds.). (2015). The global airline industry . John Wiley & Sons.

Bongo, M. F., & Ocampo, L. A. (2017). A hybrid fuzzy MCDM approach for mitigating airport congestion: A case in Ninoy Aquino International Airport. Journal of Air Transport Management, 63 , 1–16.

Burghouwt, G., & de Wit, J. (2005). Temporal configurations of European airline networks. Journal of Air Transport Management, 11 (3), 185–198.

Cacchiani, V., & Salazar-González, J. J. (2020). Heuristic approaches for flight retiming in an integrated airline scheduling problem of a regional carrier. Omega, 91 , 102028.

Chen, C. H., Chou, F. I., & Chou, J. H. (2020). Multiobjective evolutionary scheduling and rescheduling of integrated aircraft routing and crew pairing problems. IEEE Access, 8 , 35018–35030.

Chen, S. J., & Hwang, C. L. (1992). Fuzzy multiple attribute decision making methods. Fuzzy Multiple Attribute Decision Making , pp. 289–486.

Cook, G. N., & Goodwin, J. (2008). Airline networks: A comparison of hub-and-spoke and point-to-point systems. Journal of Aviation/aerospace Education and Research, 17 (2), 1.

Google Scholar  

Çiftçi, M. E., & Özkır, V. (2020). Optimising flight connection times in airline bank structure through Simulated Annealing and Tabu Search algorithms. Journal of Air Transport Management, 87 , 101858.

Deveci, M., & Demirel, N. Ç. (2018). Evolutionary algorithms for solving the airline crew pairing problem. Computers and Industrial Engineering, 115 , 389–406.

Deveci, M., Demirel, N. Ç., & Ahmetoğlu, E. (2017). Airline new route selection based on interval type-2 fuzzy MCDM: A case study of new route between Turkey-North American region destinations. Journal of Air Transport Management, 59 , 83–99.

Deveci, M., Özcan, E., John, R., & Öner, S. C. (2018). Interval type-2 hesitant fuzzy set method for improving the service quality of domestic airlines in Turkey. Journal of Air Transport Management, 69 , 83–98.

Dožić, S., Lutovac, T., & Kalić, M. (2018). Fuzzy AHP approach to passenger aircraft type selection. Journal of Air Transport Management, 68 , 165–175.

Dutta, B., Labella, Á., Rodríguez, R. M., & Martínez, L. (2019). Aggregating interrelated attributes in multi-attribute decision-making with ELICIT information based on Bonferroni mean and its variants. International Journal of Computational Intelligence Systems, 12 (2), 1179–1196.

Estrella, F. J., Espinilla, M., Herrera, F., & Martínez, L. (2014). FLINTSTONES: A fuzzy linguistic decision tools enhancement suite based on the 2-tuple linguistic model and extensions. Information Sciences, 280 , 152–170.

Görener, A., Ayvaz, B., Kuşakcı, A. O., & Altınok, E. (2017). A hybrid type-2 fuzzy based supplier performance evaluation methodology: The Turkish Airlines technic case. Applied Soft Computing, 56 , 436–445.

Herrera, F., & Martı́nez, L., & Sánchez, P. J. (2005). Managing non-homogeneous information in group decision making. European Journal of Operational Research, 166 (1), 115–132.

Ishizaka, A., & Nemery, P. (2013). Multi-criteria decision analysis: Methods and software . John Wiley & Sons.

Kahraman, C., Onar, S. C., & Oztaysi, B. (2015). Fuzzy multicriteria decision-making: A literature review. International Journal of Computational Intelligence Systems, 8 (4), 637–666.

Karaman, A. S., & Akman, E. (2018). Taking-off corporate social responsibility programs: An AHP application in airline industry. Journal of Air Transport Management, 68 , 187–197.

Kuo, M. S. (2011). A novel interval-valued fuzzy MCDM method for improving airlines’ service quality in Chinese cross-strait airlines. Transportation Research Part E: Logistics and Transportation Review, 47 (6), 1177–1193.

Kuo, M. S., & Liang, G. S. (2011). Combining VIKOR with GRA techniques to evaluate service quality of airports under fuzzy environment. Expert Systems with Applications, 38 (3), 1304–1312.

Labella, Á., Dutta, B., & Martínez, L. (2021). An optimal best-worst prioritization method under a 2-tuple linguistic environment in decision making. Computers and Industrial Engineering, 155 , 107141.

Liang, F., Brunelli, M., & Rezaei, J. (2020). Consistency issues in the best worst method: Measurements and thresholds. Omega, 96 , 102175.

Mahtani, U. S., & Garg, C. P. (2018). An analysis of key factors of financial distress in airline companies in India using fuzzy AHP framework. Transportation Research Part a: Policy and Practice, 117 , 87–102.

Palomares, I., Rodríguez, R. M., & Martínez, L. (2013). An attitude-driven web consensus support system for heterogeneous group decision making. Expert Systems with Applications, 40 (1), 139–149.

Pandey, M. M. (2020). Evaluating the strategic design parameters of airports in Thailand to meet service expectations of Low-Cost Airlines using the Fuzzy-based QFD method. Journal of Air Transport Management, 82 , 101738.

Perçin, S. (2018). Evaluating airline service quality using a combined fuzzy decision-making approach. Journal of Air Transport Management, 68 , 48–60.

Quesnel, F., Desaulniers, G., & Soumis, F. (2020). Improving air crew rostering by considering crew preferences in the crew pairing problem. Transportation Science, 54 (1), 97–114.

Rezaei, J. (2015). Best-worst multi-criteria decision-making method. Omega, 53 , 49–57.

Rezaei, J., Fahim, P. B., & Tavasszy, L. (2014). Supplier selection in the airline retail industry using a funnel methodology: Conjunctive screening method and fuzzy AHP. Expert Systems with Applications, 41 (18), 8165–8179.

Sang, X., Liu, X., & Qin, J. (2015). An analytical solution to fuzzy TOPSIS and its application in personnel selection for knowledge-intensive enterprise. Applied Soft Computing, 30 , 190–204.

Shih, H. S., Shyur, H. J., & Lee, E. S. (2007). An extension of TOPSIS for group decision making. Mathematical and Computer Modelling, 45 (7–8), 801–813.

Torlak, G., Sevkli, M., Sanal, M., & Zaim, S. (2011). Analyzing business competition by using fuzzy TOPSIS method: An example of Turkish domestic airline industry. Expert Systems with Applications, 38 (4), 3396–3406.

Triantaphyllou, E. (2000). Multi-criteria decision making methods. In Multi-criteria decision making methods: A comparative study (pp. 5–21). Springer, Boston, MA.

Tsaur, S. H., Chang, T. Y., & Yen, C. H. (2002). The evaluation of airline service quality by fuzzy MCDM. Tourism Management, 23 (2), 107–115.

Wang, Y. J., & Lee, H. S. (2007). Generalizing TOPSIS for fuzzy multiple-criteria group decision-making. Computers and Mathematics with Applications, 53 (11), 1762–1772.

Wei, K., Vaze, V., & Jacquillat, A. (2020). Airline timetable development and fleet assignment incorporating passenger choice. Transportation Science, 54 (1), 139–163.

Wu, H. Y., Chen, J. K., & Chen, I. S. (2012). Performance evaluation of aircraft maintenance staff using a fuzzy MCDM approach. International Journal of Innovative Computing, Information and Control, 8 (6), 3919–3937.

Xiao, M., Chien, S., Schonfeld, P., & Hu, D. (2020). Optimizing flight equencing and gate assignment considering terminal configuration and walking time. Journal of Air Transport Management, 86 , 101816.

Yan, C., Barnhart, C., & Vaze, V. (2020). Choice-Based Airline Schedule Design and Fleet Assignment: A Decomposition Approach. Available at SSRN 3513164.

Yazdi, A. K., Kaviani, M. A., Hanne, T., & Ramos, A. (2020). A binary differential evolution algorithm for airline revenue management: A case study. Soft Computing, 24 (18), 14221–14234.

Download references

Acknowledgements

The authors would like to thank experts from Network Planning and Scheduling Department of Turkish Airlines for the useful discussions and feedback about alternatives and ranking.

This work is partially supported by the Spanish Ministry of Economy and Competitiveness through the Spanish National Project PGC2018-099402-B-I00, the Postdoctoral fellow Ramón y Cajal (RYC-2017–21978), the FEDER-UJA project 1380637 and the Junta de Andalucía, Andalusian Plan for Research, Development, and Innovation (POST- DOC 21-00461).

Author information

Authors and affiliations.

Royal School of Mines, Imperial College London, London, SW7 2AZ, UK

Muhammet Deveci

Department of Industrial Engineering, Turkish Naval Academy, National Defence University, 34940, Tuzla, Istanbul, Turkey

Department of Computer Science, University of Jaén, 23071, Jaén, Spain

Rosa Mª Rodríguez & Álvaro Labella

Department of Industrial Engineering, Istanbul Atlas University, 34408, Kagithane, Istanbul, Turkey

Muharrem Enis Ciftci

Product Management, Ordulu Technology Inc, 06200, Ankara, Turkey

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Muhammet Deveci .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Tables 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Deveci, M., Rodríguez, R.M., Labella, Á. et al. A decision support system for reducing the strategic risk in the schedule building process for network carrier airline operations. Ann Oper Res (2022). https://doi.org/10.1007/s10479-022-04999-4

Download citation

Accepted : 15 September 2022

Published : 14 October 2022

DOI : https://doi.org/10.1007/s10479-022-04999-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Network carrier airline
• Best worst method (BMW)
• Multi-criteria decision making (MCDM)
• Find a journal
• Publish with us
• Track your research

•   Home
• University of Bedfordshire e-theses
• Masters e-theses

A decision support system for international students

Description

Collections.

The following license files are associated with this item:

entitlement

Export search results

The export option will allow you to export the current search results of the entered query to a file. Different formats are available for download. To export the items, click on the button corresponding with the preferred download format.

By default, clicking on the export buttons will result in a download of the allowed maximum amount of items.

To select a subset of the search results, click "Selective Export" button and make a selection of the items you want to export. The amount of items that can be exported at once is similarly restricted as the full export.

After making a selection, click one of the export format buttons. The amount of items that will be exported is indicated in the bubble next to export format.

• Electronic Theses and Dissertations

Files in This Item:

Google Scholar TM

Items in Digital Knowledge are protected by copyright, with all rights reserved, unless otherwise indicated.