Evaluating the Effectiveness of training
The effectiveness of any training intervention is usually measured through formal evaluation. A popular model used is that of Fitzpatrick who identified the need for this evaluation to take place at four levels. First, the immediate reaction of the trainee to the training they have received is measured. This is usually achieved through what is known colloquially as a “happy sheet” where people rate aspects of the training experience as diverse as the knowledge of the trainer and standard of handouts to the refreshments and ambience of the room.
This level of evaluation, whilst seen as having a place in the overall evaluation process, is generally not thought to be of much value. It can serve to correct relatively minor problems in the training delivery and serves as a confidence booster to the trainer(s). It is also rarely utilised in on-the-job training. The second level is that of “learning” which aims to assess what the trainees actually learnt in the programme. Measurement at this level is achieved through the use of tests or questionnaires and tends to concentrate on the aspect of retention of material rather than application. The third level seeks to determine whether the individual is applying what they have learnt when they return to the workplace. The method of measurement will depend on the type of training received.
If a process was trained, such as completing certain administrative tasks, the product of the trainee can be assessed. However, if behaviour or attitude have been trained, such as dealing with customer complaints, it can be more difficult to determine the success of the training unless the trainee is questioned as their approach. The fourth level is involved in determining whether the training intervention is having the expected impact on the organisational performance. A training need may have been identified to improve customer service and may have been well received at level one, proved to have been retained at level two, shown to have been implemented at level three, but may not result in increased satisfaction. Failure at this stage is usually due to the content of the training being inappropriate. Evaluation at this stage can be split between departmental and organisational performance thereby resulting in a fifth level (Reid et al 1994).
Compass had evaluated their Vision and Values programme and reported the results at level four. Measures of the effectiveness of the integrated training and development programme have been based around the results of staff surveys, the number of job vacancies filled by internal applicants and staff turnover figures. The results are promising, as since December 2003, 15% more vacancies have been filled internally and there has been a 5% reduction in staff turnover. It has been calculated that this has saved £1.5 million in recruitment, training and associated costs. The last staff satisfaction survey found that many employees rated this basic training package as being more important to them than higher pay (Training Magazine 2005).
The purpose of the research was to assess the effectiveness of off-the-job training against on-the-job training using the specific example of Compass Group. The approach that was taken was to determine the views of the employees of Compass Group as to the relative effectiveness of each type of training and to then assess this against how they used training they had received in the workplace. This was based on the premise that for training to be effective, it must be off use to the trainee. A certain amount of collaboration of the findings could be achieved through a discussion with one of the mobile training team – the group of people who are rolling out the Vision and Values programme. The common method of measuring the effectiveness of a training intervention is through the four stages of evaluation, so the person responsible for training should also be questioned on how they measure effectiveness to see if empirical information could be collected. Compass Group were approached to provide more detailed evaluation results, but declined.
Several areas became apparent in the literature review which would need to be resolved in the questionnaire for the results to be of value. Firstly, the management and non management employees needed to be asked different sets of questions. The management population would be expected to hold views on the effectiveness of the on-the-job and off-the-job training that they, or their colleagues carried out with employees but would also have an opinion on their own feelings towards different training methods that they experienced. It was also important to question a person responsible for training and development to ascertain the organisations policies and approaches to training and development and the reasons behind this. As mentioned, although the head office personnel declined to participate in the research, one member of the mobile training team who was in one of the units taking part in the survey, agreed to a short interview that provided useful supporting information.
Whilst the Compass Group employs personnel across a wide range of disciplines, for the purpose of this report, the emphasis was on front-line professionals working within two specific units. These roles include, waiting staff, general assistants and catering management. Higher levels of management were not included in the research, nor were head office personnel, with the exception of the brief interview with one of the twelve members of the mobile training team.
The research was conducted in two Compass Group run staff restaurants. The total number of respondents was twenty-eight which was made up of four managers, three waiting staff, sixteen general catering assistants, four till operatives and one member of the mobile training team. The survey was a combination of structured questions and informal interviews. This was done to initially ascertain their views on training and then to be able to probe more deeply to overcome the potential under appreciation of on-the-job training that was discovered during the literature review. The formal questions were in two sets, one of which was asked only of the managers to determine their views on the training their staff received, whilst the second set was asked of everyone including the managers to ascertain their views on the training they had received. The member of the mobile training team had a separate, short interview with the writer.
The questions asked of both the management and non management groups were designed to collect their views and beliefs on the amount and relative effectiveness of on-job versus off-job training. The information gathered was based purely on their perception, as they were not asked to refer to training records or evaluations. The first set of questions were generally closed questions some of which utilised the Likert Scale.
The final two questions of the formal questionnaire were seen as being the key to measuring the relative effectiveness of each dichotomy of training delivery. The writer assessed whether the answer the person gave was either a skill and /or behaviour or related to attitude or knowledge. The respondent was then asked to confirm the classification of their answer. They were then asked how this training had been delivered. Again, their reply was classified as on-the-job or off-the-job by the writer to ensure consistent definition which, as mentioned in the literature review, can be difficult. This helped lead them into the informal interview stage of the survey as it encouraged them to think about the relative importance of each type of training delivery. During this stage, the questions were open questions designed to collect views and opinions. For this part of the research, the results are shown with examples of the most popular comments given. Before the final two questions were asked, the researcher gave a brief explanation as to the four levels of evaluation and asked them to comment against each. This was seen as the most effective way of gaining their opinion rather than the researcher classifying general comments into the four levels.
Potential limitations of the survey were assessed. It was identified by the writer that the survey population was relatively small (twenty eight) and that this may result in unrepresentative results for the organisation as a whole. Furthermore, the emphasis of the survey was initially to obtain their perspective of the relative effectiveness of on-the-job versus off-the-job training interventions and by definition, perception may differ from reality. The writer believe they overcame this potential limitation by extending the survey to assess how the training they found of most value to them in their jobs was delivered to them and from this, to determine the relative effectiveness of each type of delivery through the application of the training to their jobs. The writer believes that this is an acceptable form of the level four type of evaluation described earlier.
Tags: attitude, confidence booster, customer service, dealing with customer complaints, formal evaluation, organisational performance
