«Using EFQM in higher education: Ten years of experience with programme auditing at Hanzehogeschool Groningen Evert Bisschop Boele, Hiltje Burgler, ...»
Hanzehogeschool Groningen works with an internal quality assurance system that has been developed over the past decade and a half. The work started in the early nineties when quality assurance experts from the Hogeschool van Amsterdam, Hogeschool InHolland, Fontys Hogescholen and the Hanzehogeschool Groningen gathered in the informal HBO expert group (‘Expertgroup HBO’; HBO stands for hoger beroepsonderwijs, higher professional education, the type of education provided by hogescholen in the Netherlands). The goal of the HBO expert group, associated member of the European Foundation for Quality Management, was “the exchange and dissemination of knowledge and experience in the field of Total Quality Management in higher education” (Expertgroup HBO, 2006, 1).
On their search for a suitable quality assurance model for higher education institutions, they opted for the EFQM/INK model. INK stands for the Dutch Quality Institute (Instituut Nederlandse Kwaliteit). The EFQM (European Foundation for Quality Management) model basically looks at an organisation, its results, and the way the results lead to learning, improvement and innovation. It was developed for firms but can be applied to any kind of organisation. The model is visualised in the following well-known scheme with nine so-called “criteria” (see Graph 1).
The graph displays the organisational processes (criteria 1–5) at the left and the results (criteria 6–9) at the right hand side, which are subject to evaluation and measurement.
Where necessary, adjustments are made on the basis of these assessments, thereby creating a coherent feedback loop. The central criterion in the EFQM model is number five, “management of processes”. In the case of higher education institutions, this may refer to education and/or research.
The Dutch Quality Institute INK extended the EFQM model by adding five developmental stages (see Tillema/Markerink 2004, p. 13–30). For each of the nine criteria, the organisation can be scored on a scale ranging from one to five, corresponding to the five developmental stages and expressing how sure it can be of obtaining quality regarding the
respective criterion. The five developmental stages can be shortly described as follows:
● Stage 1 – Activity oriented: All activities take place on an individual and ad-hoc basis without written documentation;
● Stage 2 – Process oriented: Processes, teamwork and documentation are to some degree established;
● Stage 3 – System oriented: Processes are systematically improved on the basis of evaluation;
Beiträge zur Hochschulforschung, Heft 1, 30. Jahrgang, 2008 Evert Bisschop Boele, Hiltje Burgler, Henriette Kuiper ● Stage 4 – Chain oriented: External orientation (towards suppliers and customers; in the case of higher professional education for example secondary schools and labour market respectively) is broadly embedded in the organisation;
● Stage 5 – Total Quality Management: The organisation is innovative and takes initiatives, recognises its role in society, and benchmarks itself against other excellent organisations.
Growth from stage one to stage five usually takes place when shifts in the following
dimensions take place:
● orientation: from internal to external, ● participation: from individual to society, ● policy making: from ad hoc to systematic, ● documentation: from informal to well-established, ● improvement: from problem-solving to anticipatory.
This can be graphically depicted as follows:
Graph 2: Dimensions of growth from stage 1 to stage 5
The HBO expert group developed a specification of the EFQM/INK model for universities of applied sciences called “Method for Improving the Quality of Higher Education Based on the EFQM model” (Expertgroep HBO, 1995). This version was incrementally refined on the basis of the findings from audits and in response to external developments. The newest update (fourth edition) takes into account the current accreditation system (Expertgroep HBO, 2004). The publication is in great demand and available in English language (Expertgroup HBO, 2006).
5 Programme auditing at Hanzehogeschool Groningen How does auditing at Hanzehogeschool Groningen work? At Hanzehogeschool Groningen auditing at programme level takes place according to a three-year cycle. Sometimes, closely related programmes are audited together. Basically the auditing process consists of two phases: self evaluation followed by external evaluation. Different from many quality assurance systems, the self evaluation does not lead to the publication of a selfevaluation report. Instead, its aim is to reach consensus on a table of scores among those involved in the self evaluation.
The audit process starts with an instruction meeting in which everyone involved in the self evaluation phase of the auditing process is explained by an expert how the EFQM model works. It differs from programme to programme who will be involved, as this is the free choice of the programme management. It may be only the programme management, but mostly the key – or sometimes all – teachers are involved, often people from support services such as secretaries, planners or people concerned with the teaching facilities, and sometimes students. The number of people involved may range from five to more then twenty.
Every person involved then independently scores the programme along the nine criteria (all sub-divided into several aspects) of the EFQM model on a separate form. These scores are collected and put together in one overview, which is discussed in a so-called consensus meeting. During this meeting, members of the team present arguments for their scores and try to reach consensus. In most cases, the consensus meeting is led by a chair external to the programme, coming from the pool of trained auditors from Hanzehogeschool Groningen, who prepares a short consensus report. It consists of the scores and short explanations along the different criteria and aspects, and completes the internal phase. On the basis of this report, the programme management can draw conclusions on perceived strengths and weaknesses and make a first inventory of points to improve in the future.
Next, an external committee of three or four trained auditors is put together. The committee members are external from the point of view of the programme, but come from inside the institution. Based on a set of documents compiled by the programme team on demand, they score the programme provisionally. This is followed by the actual audit, a one-day visit in which the auditors seek confirmation of their initial scores, ask for explanation etc. Towards this end, they hold interview sessions with the management, Beiträge zur Hochschulforschung, Heft 1, 30. Jahrgang, 2008 Evert Bisschop Boele, Hiltje Burgler, Henriette Kuiper teachers, students, support staff, external stakeholders and a member of the executive board of the institution. Some of them will probably have been involved in the self evaluation phase of the audit, but this is no necessity. Based on these meetings, the committee compiles a final report with recommendations. It is handed over to the programme; a copy goes to the executive board. This document again gives the programme management suggestions for points of improvement, and comparison of the outcomes of the internal and the external phase of the audit may shed light on possible blind spots in the programme’s self-perception.
This process is repeated once every three years for every programme, a frequency that has been set in line with the six-year rhythm of external accreditation. A “preparatory audit” is run about a year before the site visit of the accreditation agency takes place, an “intermediate audit” three years later. Programmes tend to value the preparatory audit highly, as it is perceived as a useful preparation for the accreditation process. The character of audit versus accreditation is nevertheless quite different: audits are process and improvement focused whereas accreditations are more product-oriented and lead to a yes-/no decision. From the perspective of quality assurance, the intermediate audit is no less important. Two years after accreditation, the concern for the quality of processes may have faded away a bit; an audit can then help to re-establish the focus on quality orientation among those involved in the running of the programme.
The audits are performed by trained auditors. Many of them are teachers, quite a few come from the support staff units, and management (some deans of schools, but mainly the management layer just below, so-called “team leaders” who are directly responsible for day to day business) is represented as well. At present, there are about eighty trained auditors in the institution, of which about fifteen have received more intense training in order to be able to act as chairperson of an audit committee. The training is organised internally, but the quality of the training is assured by the fact that many of the trainers have received external training on the EFQM/INK methodology.
6 Effects of the audit system
Does the audit system lead to measurable improvement? This is hard to judge as no clear criteria are available. No methodologically sound ranking system exists for Dutch universities of applied sciences which could be used to measure improvement. But even if there was, hard scientific proof of a causal link between the audit system and positive ranking results would be hard to establish. The same counts for the results of our own 102 Beiträge zur Hochschulforschung, Heft 1, 30. Jahrgang, 2008 Hanzehogeschool Groningen planning and control cycle: although the university is performing well in many aspects, there is no possibility to link this one-to-one to the use of the EFQM auditing system.
Comparison with other universities of applied sciences is no option either as there are no institutions with a fully comparable system. Our main body of evidence consists of about eighty audit reports, gathered over a period of about ten years, which depict changes regarding both the internal and the external situation.
A global study of this body of evidence as well as personal observation suggests that the appreciation of internal quality assurance has grown over the years through participation of a wide range of members of the institution in the programme audits. For what it is worth: average scores in audit reports are steadily mounting, from an overall score near stage two to an overall score well between stages two and three. In the external accreditation reports, the internal quality assurance system consistently receives a “good” score (on a four point scale ranging from “insufficient” through “sufficient” and “good” to “excellent”). At a conference at Hanzehogeschool Groningen on 12 October 2006, when we celebrated the fact that we have been using EFQM audits for a period of ten years, the president of the Dutch-Flemish Accreditation Organisation NVAO expressed his satisfaction with the work the Hanzehogeschool Groningen has done. We therefore carry on with the audit system. The growing appreciation of the importance of internal quality assurance alone would provide sufficient motivation.
The many audits and consensus meetings at Hanzehogeschool Groningen convinced us that quality assurance is not only concerned with measurement, but also with the processes in which many of our colleagues are actively involved. The approach is a uniform one, but variants exist in order to meet the requirements of specific programmes. The scope of audit may range from one to two or three programmes, the definition of “leadership” may vary from programme to programme (e. g. dean, team leader, or management team), sometimes the organisational criteria 1–5 are discussed with teachers and students whereas the results criteria 6–9 are discussed only with management, et cetera.