ISSN 0798 1015

Vol. 40 (Number 01) Year 2019. Page 29

Evaluation model for effectiveness of corporate training based on Key Performance Indicators and detailed metrics

Sistema de evaluacion para medir la efectividad del aprendizaje corporativo basado en Key Performance Indicators y métricas detalladas

Konstantin K. ILYUSHNIKOV 1; Svetlana V. LOBOVA

Received: 03/08/2018 • Approved: 16/11/2018 • Published 14/01/2019

Contents

1. Introduction

2. Methods and data

3. Results

4. Conclusions

References

1. Introduction

The growth of innovative processes in the economic sphere entails an increase in the requirements for the professional training of specialists, concurrently, being accompanied by an acceleration of the regular loss of relevance and practical applicability of knowledge (Paper, 1998).

Since the 1970s, the in-house training of employees has become the main tool for basic training and updating the knowledge and skills of employees. With wide spreading of corporate universities around the world, the issue becomes topical as regards the methods for assessing the professional training of personnel, their applicability to the specifics of training, reliability and correlation with the key performance indicators of an organization.

The objective of the article is to study current theoretical trends pertaining to the corporate training effectiveness evaluation; to develop a practical model of the evaluation based on the KPIs and detailed metrics; to describe the tools of its implementation in an organization.

The article has the following structure: Introduction presents topicality of the issue in question, describes the structure and objective of the paper; Part I describes the evolution of training effectiveness evaluation methods and modern trends of the evaluation approaches; Part II describes the training effectiveness evaluation model based on KPIs and detailed metrics of an organization; Part III includes tools for the model implementation; Conclusion gives general findings as regards the issue in question.

For more than 50 years of development of evaluation methods, a variety of methods have been created, each of which corresponded to the development trends of corporate universities. In order to determine the current vector of development of forms and methods of the evaluation, below the main steps are described:

Basic (1948 – 1973): formation of major evaluation approaches and principles. The suggested theories lack an urge to evaluate the training effectiveness by the finance criterion. Within this period the following models were developed: Taxonomy of Educational Objectives (Bloom, 1965), CIPP (Stufflebeam, 2000), UCLA (Alkin, 1969), CIRO (Warr, 1970).

Genesis (1974 – 1990): classical approaches developed at the previous stage and, sometimes, seeming to be a detailed concept, acquire a status of major theories; endure over time and consolidate their position in the academic and professional communities. The following approaches emerge: Deductive Approach (Hamblyn, 1974), Four-Level Training Evaluation Model (Kirkpatrick, 2004), Naturalistic Approach (Guba, 1978), Calculation of Costs (Fitts-ens, 1984)

Dissemination (1991 – 2009): wide spreading and practical implementation of theoretic approaches to effectiveness evaluation. Increase in number of corporate universities results in the need for calculation of return on investment (ROI). Significance (or even supremacy according to some new theories) is assigned to the training payback criterion. During this period, the following methods appear: ROI (Phillips, 1997), Balanced Score (Kaplan and Norton, 2010), Five OEM-Based Levels (Kauffmann, 1995), Targeted Approach (Tyler, 1998), Expert Reviews (Eisner, 1998), Anderson and Krathwohl Taxonomy (Anderson and Krathwohl, 2001);

Integration (2009 – till present): evaluation of effectiveness of the training system goes to the level of business indicators. HR units get objectives identical to those of KPIs of major units of an origination. At present, the approach evolution has reached the stage where the training effectiveness is evaluated by the growth of the company’s internal KPIs, therefore, having generated the following thesis: “Education is a success when it facilitates the growth of company’s key performance indicators”, although, currently, there are no methods for the evaluation of training effectiveness that would meet this field.

As a result of our research of theoretical aspects and evolution of the corporate training effectiveness evaluation, we have found out that the modern trends (Jennings, 2014; Beer, 2016) implicate the focus of the evaluation system on the growth of company’s KPIs. According to this concept, developmental activities performed in a company cannot be evaluated without a detailed system for KPIs. Besides, we haven’t managed to find any real-life described examples of complete integration of KPIs to the organization of training and effectiveness evaluation. The above authors describe a theoretical part of the model as follows: “If the performed training has exceeded one of the KPIs, it means that it is effective” (Beer, 2016).

3.1 Development of KPI-Based Corporate Training Effectiveness Evaluation Model.

KPI & Metrics-based model for the evaluation of the corporate training effectiveness will have the dynamics of the company’s key performance indicators as the highest priority which not only performs control function, but also training planning one (effective training starts at the planning stage). When training the employees who do not need training, no significant increase in metrics and KPIs can be observed (such employees meet them before the training), therefore, such training is not effective meaning that the evaluation model should also account for the planning process as well.

In order to implement such a model in an organization, the following elements should be in place:

1. Availability and practical use of a detailed system of key performance indicators (KPIs);
2. Metrics system within each KPI (component parts meeting the KPIs depend on);
3. Reporting system for each of the KPIs updated on a regular basis in real time;

Let us proceed with the detailed reporting system for KPIs and metrics within the remedial training course. The main task of the system at this stage is to define the indicators that require improvement and the operators the developmental activities will be addressed to. Without such an approach, planning of employees requiring the training is performed by their superior or the trainers themselves majorly based on the observation method. In some cases the training planning is based on a “train everybody” principle irrespective of the area and topics of the trainings, in which case the training should also be taken by the employees who will not benefit from it. At the same time, the company spends the resources for the employees’ downtime, involvement of the trainers, stationaries, etc. Consider how a model of remedial training should look like when the Detailed KPIs and Metrics Reporting System applies, as shown in Figure 1.

Figure 1
Model of Planning and Evaluation of Remedial Training Based on Employees’ KPIs and Metrics

Source: compiled by the author

3.2 Practical Approbation of KPI-Based Corporate Training Effectiveness Evaluation Model.

Consider the algorithm for selecting employees for remedial training using the example of a group of 11 employees, with 5 metrics for each (for convenience of demonstration, employees who failed to meet the target value by metrics are highlighted).

Table 1
KPIs and Metrics of the Pilot Group after the Implementation of
MAP-Based Effectiveness Evaluation Model / Before Remedial Training.

Source: compiled by the author

To identify the employee in need of the training, we shall use the Trend principle, i.e. failure to meet the metrics two months in a row. Based thereupon, we are going to exclude the elements of chance, season, temporary decrease in motivation and ability to analyze and increase the indicator independently and/or together with a superior. Should the same employee fail to meet the target value several times, he or she will be included to the necessary training course.

Take a look at Table 1: Employees 2; 4; 6; 8; 11 have failed to meet the necessary metrics values, and will, therefore, be scheduled to the training. To do so, in order to increase each of the metrics the company should generate a toolkit of developmental activities suitable for both on-site, and off-site (distant training) forms for the convenience of the employees and the management. Thus, necessary developmental activities will be scheduled for the entire personnel on a monthly basis.

We believe that it would appropriate to group the metrics with similar focuses to perform general training event, which would eliminate the need for multiple local training events. In this example, to satisfy the need for developing 6 metrics, 2 seminars have been selected (Table 2).

Table 2
Metrics-Based Employees Scheduling for Seminars

Source: compiled by the author

As a result of dividing the KPIs to the metrics, training specialists will get a detailed insight of all component parts needed to meet company’s KPI. For instance, to meet the servicing unit’s KPI “resolving a customer’s issue within one visit”, an employee should demonstrate an adequate level of proficiency as regards the metrics from Table 2. Criteria that meet the standards of performing the jobs and, when successfully followed, would results in employee meeting the KPIs are selected as the metrics. Thus, the trainers develop criteria (metrics) that are significant for the company and that directly influence meeting the KPIs.

After training, which objective was to increase the KPIs, we have re-measured the KPIs and metrics values (Table 3).

Table 3
KPIs and Metrics of the Pilot Group after the Implementation of
MAP-Based Effectiveness Evaluation Model / after Remedial Training

Source: compiled by the author

Out of 5 employees who initially failed to meet the target values, three have met the target values, while the other two have improved their results significantly: Employee 2 from 69% to 83%, and Employee 4 from 57% to 75%. The results of the pilot group make it possible to conclude that, at the stage of the remedial training, the MAP-based model enables scheduling employees who fail to meet the metrics, establishing an appropriate developmental activity for them, and that in case of positive post-training dynamics of the metrics the training can be deemed effective.

For the sake of convenience of analyzing a data set, employees may be scheduled for training not through the manual search, but rather through the automatic one using macros for group generation. When implementing such model, the need for training will be checked at two levels: on the one hand, the employees requiring remedial training will be identified; on the other hand, topics in which such employees require training will be displayed automatically.

Our MAP-Based Corporate Training Effectiveness Evaluation Model using KPIs and detailed metrics meets contemporary world trends in the area, and sets monitoring and evaluation of the company’s key performance indicators as its goal, as well as ensures their continuous improvement through the planning of developmental activities. Dividing of the KPIs to metrics provides local and addressed monitoring of employees’ performance within the KPIs. To increase training effectiveness, planning of the remedial training (including establishment of the developmental activities) should be commensurate with the metrics measurement results and their analysis. Also, the remedial training is the most effective when an employee shows systemic failure to meet an index according to the reports.

The development of the KPIs & metrics-based effectiveness evaluation system does not substitute the use of conventional evaluation levels, such as knowledge assessment (examinations, tests, etc.), expert review of trainers’ work, etc. But assigning the highest priority to the evaluation of training effectiveness by key performance indicators of the entire company creates a vector aimed at the true purpose of the training – assist the business in achieving its goals. At the same time, there are trainings, effectiveness of which does not need to be assessed using the KPIs and Metrics system (e.g., fire safety). The assessment of theoretical knowledge and/or practical skills test should suffice.

Alkin, M.C. Evaluation Theory development. Evaluation Comment 2.2-7.(1969)

Anderson, L., and Krathwohl, D. A. Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman.(2001).

Beer,  M. The Great Training Robbery 4/21/2016 By Michael Beer Harvard Business School & Magnus Finnstrom & Derek Schrader URL: http://www.hbs.edu/faculty/Publication%20Files/16-121_bc0f03ce-27de-4479-a90e-9d78b8da7b67.pdf (2016)

Bloom, B. and Englehart, M. and Furst, E., and Hill, W., and Krathwohl, D. Taxonomy of educational objectives: The classification of educational goals. HandbookI: Cognitivedomain. NewYork, Toronto: Longmans,Green.(1956)

Eisner, E. W. The enlightened eye: Qualitative inquiry and the enhancement of educational practice Upper Saddle River, NJ: Prentice-Hall. – (1998).

Fitz - enz, J. How to measure human resources management. New York: McGraw-Hill.(1984)

Guba, E. G. Toward a methodology of naturalistic in-quiry in educational evaluation / E. G. Guba // CSE Monograph Series in Education. – Vol.8. – Los Angeles: Center for the Study of Evaluation, - (1978)

Hamblin, A C. Evaluation and Control of Training, McGraw Hill (1974)

Jennings, C. Focused on all things related to learning, performance and organizational productivity, and to the 70:20:10 model. (2014).

Kaplan, R.  Conceptual Foundations of Balanced Scorecard Harvard Business School, Harvard University (2010)

Kaufman, R., and  Keller, J., and Watkins, R. What works and what doesn’t: Evaluation beyond Kirkpatrick. Performance and Instruction, 35(2): 8-12. (1995)

Kirсpatrick, D. AT+D classic: how to start an objective evaluation of your training program T&D, May, v5 i5.(2004)

Paper, W. Employment for the 1990-s. //White Paper L. P.12 (1998)

Phillips, J. Handbookof Training Evaluation and Measurement Methods / J. J. Phillips. – 3nd ed. – GulfProfes-sional Publishing, 1997. – 420 c.

Stufflebeam, D. L. The CIPP modelforevaluation / D. L. Stufflebeam, G. F. Madaus, T. Kellaghan (Eds.) // Evaluation models. – 2nded. – Boston: Kluwer Academic Publishers, (2000)

Tyler, R. The Basic Principles of Curriculum and Instruction / R. Tyler. – Ornste in and Hunkins,– 128c. (1998)

Waddington, T.  and Aaron, B. and Sheldrick, R - Guerilla Evaluation: Adapting to the Terrain and Situaion..- Instructional Desing in the Real World: A View from the Trenches, - 271 с.(2004)

Warr, P., and Bird, M., and Rackham, N. Evaluation of Management Training. London Gower Press. – (1970).

1. Ural State University of Economics, Ekaterinburg, Russia. Email: barnaulhome@mail.ru

2. Altai State University, Barnaul, Russia