Information system for academic staff performance evaluation

Description of the IS HAP

The intention to create a mathematical model for academic staff performance assessment was formulated at Palacký University, Faculty of Science as early as 2006. Since then various academic staff evaluation models currently used on universities both in the Czech Republic and abroad were examined. These models were subjected to a detailed analysis regarding their practical and mathematical aspects. The analysis resulted in the design of several academic staff evaluation models, differing in how members of academic staff are evaluated in separate areas of their activity and in the aggregation method for these partial evaluations (weighted average, OWA, WOWA). The final model implemented in IS HAP relies on a fuzzy expert system for the aggregation of partial evaluations.

A part of the form filled in by each staff member. Evaluation of a particular staff member. Evaluation overview for more academic staff members.

Screenshots from the software implementation - the filled in form (left), detailed evaluation of particular academic staff member (center), staff members evaluations overview (right).

General requirements on the model were as follows: It should (1) include, if possible, every aspect of academic staff activity; (2) use only easy to verify and objective data; and (3) be easy to work with. Other requirements were for the final evaluation: (4) to maximally reflect staff benefit to the university; and (5) not to be a simple average of partial evaluations in separate areas of activity, but to be able to flexibly respond to management needs. The desired output of the model was not to arrange members of academic staff in order of their performance, nor to obtain a single crisp number interpretable only with difficulty. A basic piece of information on both focus and performance of the academic staff was considered sufficient. Such assignment implied the use of linguistic fuzzy modelling – linguistic variables, rule bases, and approximate reasoning (i.e. of fuzzy expert systems).

In the model, the performance of each member of academic staff is evaluated in both pedagogical, and research and development (R&D) areas of activities. Input data are imported from various components of the university information system into a web-based form, which is subsequently completed by the staff members by pieces of information which are not available in the university information system (yet are still relevant for the evaluation purposes). Each item in the questionnaire (an activity or the output of a staff activity) is assigned a score according to its importance or time-consumption. Three areas are taken into consideration for pedagogical performance evaluation: (a) lecturing, (b) supervising students, and (c) work associated with the development of study programs. The evaluation of research and development activities is based on the methodology for R&D outputs evaluation valid in the Czech Republic, which puts emphasis mainly on papers published in journals indexed in respected databases (WoS, Scopus), and on patents in the applied research area. Other important activities (grant project management, editorial board memberships, promotion of the field of science etc.) are also reflected in the process of R&D performance evaluation. Both pedagogical and R&D areas are assigned a standard score – different for assistants, senior assistant professors, associate professors, and professors. The number representing a partial evaluation of a member of academic staff in a certain area is determined as a multiple of the respective standard for his or her position. For better clarity and easier interpretation, these numbers are transformed into verbal evaluation using linguistic scales.

A linguistic fuzzy expert system is therefore used to aggregate both partial evaluations – for pedagogical and R&D areas of activities. The main advantage of this type of aggregation is that it allows to set-up the shape of the aggregation function completely in line with the evaluator’s requirements (e.g. to appreciate excellence achieved in one of the areas). This type of aggregation is transparent and comprehensible even to laymen as it is described in linguistic terms. The overall aggregated evaluation is also available as a linguistic expression.

The model also takes into account the load of secretarial and managerial activities with each member of academic staff (understood here as activities draining away from his or her time and thus possibly reducing the performance in each of the two areas of evaluation mentioned above). Another fuzzy expert system is therefore used to adjust the evaluation according to the managerial activities load of the particular academic staff member. The overall work load of members of academic staff is thus described in words.

After its pilot testing on one department in 2009, the described model was presented to the academic senate of the Faculty of Science, which accepted it well. An extensive pilot testing on 14 departments followed in 2010. The results of the pilot testing confirmed that the model meets all the requirements. It was therefore decided to implement the designed methodology of evaluation into an information system for academic staff performance evaluation (IS HAP). Since 2012 IS HAP has become the annual performance evaluation tool for academic staff members of the Faculty of Science of Palacký University, Olomouc. In 2016 it has been decided that IS HAP will be used on all the faculties of Palacký University and that it will constitute an important part of the information system Quality which is intended to serve for quality assurance and evaluation at this university.

IS HAP is a product of an interdisciplinary cooperation (mathematicians, computer scientists, a psychologist, a HR practitioner, and the management of the Faculty of Science participated during the development of the presented tool). This makes it a tool reflecting the needs of correctness and soundness of the used mathematical methods, but also the needs of easy interpretability and comprehensiveness. This way IS HAP is an effective support tool for human resource management.

Based on the experience and positive results achieved at the Faculty of Science of Palacký University, the pilot testing of IS HAP has been incorporated into the Individual National Project “Maintaining and assessing quality in tertiary education” in 2012-2013. IS HAP is currently being used by faculties of four other Czech universities – University of Economics, Prague, University of South Bohemia in České Budějovice, Jan Evangelista Purkyně University in Ústí nad Labem and Mendel University in Brno.

Characteristics of IS HAP

The IS HAP is a web based application, which is accessible through Internet. Each of members of the academic staff fills in a form summarizing his or her activities in the last year (Fig. 1). The form was implemented so that it would be as easy to fill it in as possible. All items of the form are divided into categories and subcategories. A brief help for any of the items is also available.

One of the requirements on the software implementation was that it must be easy to integrate with external databases and other information systems. The current version of the software is able to communicate with the information system of Palacký University in Olomouc – this way data on teaching and supervision, and the outputs of R&D are directly imported into IS HAP database. This reduces the time necessary to complete the form. IS HAP is designed to be able to communicate with and import data from various databases and information systems through web services.

Each of the academic staff members can see his or her own evaluation (Fig. 2) before the form is sent. They have also access to this form later, so the data once filled in can be used later e.g. for purposes of writing a professional CV.

After the form is filled in, the head of the respective department and the dean can see the overview of the resulting evaluations of his/her subordinate academic staff members (Fig. 3). Even though the mathematical apparatus used to calculate the evaluations is sophisticated, the results are presented in way comprehendible even to a layman, i.e. verbally and graphically (each of the performance classes is represented by a different colour). This was enabled by the use of linguistic fuzzy modelling and fuzzy classification. All the filled in forms are also available to the heads of departments (can access the forms of the academic staff members of his/her department) and to the dean (can access the forms of all academic staff members of the faculty). All these features make the resulting evaluations very easy to interpret and compare.

The head of the department is provided with the following data about members of his or her department: (1) the name of the academic staff member and his or her position, (2) evaluation of the pedagogical activities including the information on the type of the activities for which it was acquired, (3) evaluation of the research and development, which is again supplemented by an additional information on type of activities, (4) the overall evaluation in the areas of pedagogical activities and R&D, (5) the list of managerial and secretarial functions, (6) the final evaluation of pedagogical activities and R&D which also takes into the account the working time drained by the previously mentioned functions. Besides the table summarizing the results of the evaluation, the heads of the departments can view the complete forms of the members of their departments and see all of their activities. In the form, the academic staff members also specify their plans in pedagogical area and R&D for the next evaluation period. The heads of their departments can easily check if they managed to realize the plans and input a narrative evaluation report.

IS HAP represents a unique software tool. It is easy to use for both academic staff members and their superordinates. The sophisticated mathematical background provides evaluations highly-valued for the HR management, which are presented in a well-arranged and comprehensible way.

Most relevant resources