|
|
|
| Partners | Contact | ||||||||||||||||||||||||||||
|
 |
Project: Rank the rankings: a meta-analysis of public sector benchmarks (2007)
FORMULATION OF THE RESEARCH ISSUE International organizations and consultants compile a wide range of international rankings. Those comparisons concern both policy performance (environment, economy, mobility, health care, …) as the performance of the public administration (governance indicators, trust in government, e-government benchmarking, competitiveness indices). The media make news of these rankings and ministers use them to elaborate policy plans. International comparative indicators may well be a central aspect of good policy-making. Thanks to these benchmarks, Flanders can be positioned internationally. However, there are significant problems with rankings too. Mostly, the figures are accepted without any critical analysis, although regularly there are problems with data quality (Van De Walle et al. 2004). This study will look behind the figures. We will analyze the quality of the rankings, the producers and the consumers. Who collects which figures, why do they collect them and what are the problems with data quality? In other words, we aspire to rank the rankings. The main objectives of this research project are twofold. Firstly, we want to provide a judgement on the supply side. We will determine which rankings are (in)sufficient and provide an instruction leaflet for every ranking to ensure government can optimally use the right comparisons and reject insufficient rankings. Secondly, we will conduct a gap analysis on the fine-tuning of supply (what is available) and demand (what policy needs). The topic of international rankings has attracted a lot of international attention. Oxford University has organized the conference “Where does Britain rank? International Public Service Rankings” (London 13/12/2005). One of the conclusions was that a critical meta-analysis of rankings is missing. Rankings, both at the international and the national level, are also studied by the Public Services Programme of the UK Economic and Social Research Council. Moreover, the Directorate for Public Governance and Territorial Development of the OECD is conducting the study “government at a glance” for which we carried out some preparatory research (Van Dooren et al 2006). This B-project provides an opportunity to further institutionalize the connection between Flanders (SBOV) and the OECD. Finally, EGPA (the European Group of Public Administration) will study this issue for the following three years in a Study Group. This forum will enhance the international visibility of the research results. FUNDAMENTAL AND APPLIED RESEARCH QUESTIONS 1. Who produces the rankings (government, university, consultants?) Theoretical framework This study is strongly related to the research line on quantitative and prospective policy information and builds on theoretical developments in that project. The research studies the way in which information and knowledge are created, spread and used in policy-making. We will combine knowledge utilization theories with policy-oriented theories (Weiss 1980; Innes 1990). With this double perspective, we will be capable of examining knowledge while taking the uniqueness of policy processes into consideration. Research subject This B-project studies one type of information in particular, i.e. international rankings based on quantitative indicators. Primarily, we will address rankings of the apparatus of government, rather than rankings of policy sectors. Examples of research object are the governance indicators of the World Bank and surveys on trust (Van de Walle et al. 2005), general interest or customer satisfaction (European Commission, 2005), etc. Methodology Our primary source will be the reports and websites of the organizations that produce rankings. Additionally, we will search for secondary literature. Studies about internal rankings such as the many League Tables in Anglo-Saxon countries (bv. Gormley and Weimer 1999) can be interesting sources of information in order to conceptually develop metafiles. Additionally, we will look for critical research material on the issue of international rankings itself. Selectively, we will conduct some interviews with key witnesses. Metafiles A central aspect of our research design is the development of metafiles. We will make an analysis of the objectives and procedures of the rankings. Such analysis offers essential background information to determine the meaning of specific rankings. The main emphasis of the files is however on the quality of the ranking. By quality we mean the methodological quality, in which we will assess validity (do they measure the right things?) and reliability (do they measure those things right?) of the indicators, data collection procedures and analysis techniques. Another crucial aspect is the unit of analysis (regions or countries). Finally, we will also examine the quality of reporting and the transparency of the diffusion of the results. First, we will publicize reports according to the publication strategy of the “Policy Research Centre - Governmental organization in Flanders”. Secondly, we will organize several seminars. The Public Management Institute has already planned an exploratory seminar “Flanders ranked” on the 1st of December 2006, which could be arranged every two year to discuss international rankings and the position of Flanders. Furthermore, the metafiles will be made available on the internet, creating a valuable source of information for policy makers and media. Finally, the project will be a leverage to strengthen the band with international networks, the OECD in particular. 1 Development metafiles and scoring systems KU Leuven, Public Management
Institute |
|||||||||||||||||||||||||||
|
|
|
With support of the Flemish Government | Copyright © KU Leuven | Realisation: Anita Van Gils | Last change: December 19, 2006 | http://www.steunpuntbov.be |
|
