When conducting statistical comparisons of regions or countries, usually several, sometimes (for example within the competitiveness rankings of the IMD and the WEF), even several hundred indicators are used. Since the investigated units perform differently in the various characteristic values, the aggregation of individual results to an overall statement is a regular problem within these statistical comparisons.
A popular approach to the solution is to design an index based on a given number of indicators. This, for example, could be an innovation index, which provides an adequate picture of the innovation capacity of the analysed region(s) by using specific meaningful indicators. With the help of this, in particular, an appropriate comparison of individual regions can be conducted and provided.
Before the various indicators can flow into an index, they must be standardized because of their different scales. For this purpose, a number of methods are available, for example the z-transformation, in which the average value is subtracted from the characteristic value of a region and the difference is divided by the standard deviation as measure of dispersion.
The indicators which are relevant for a given topic are then often classified into indicator groups. In doing so, the individual indicator groups are usually combined additively or multiplicatively with one another, which can lead to considerable differences in the results. In case of the multiplicative combination, for example, a very low value for a single indicator or a single indicator group can restrain the value of the overall index significantly.
However, the meaningfulness of such indices is particularly sensitive to the selection of the weighting vector. Here, there is generally the risk of “Rank Engineering”, which means the deliberately controlled determination of the weighting vectors. By using methods of forensic statistics, such influence can be detected. In order to check whether and to what extend the choice of a specific weighting vector distorts the overall result, the characteristic values of the individual indicators and the overall result have to be known.
Due to the vulnerability of the classical approach regarding subjective impacts, the Cognion Research Association follows a completely different approach within its index-based benchmarking. Instead of applying exogenous weighting vectors (for examples those created by an expert committee), so-called fairness criteria and/or optimality criteria are defined and operationalized. Based on this, a weighting vector can be derived algorithmically. This weighting vector is deprived of any influence on its individual factors (for example, in the knowledge of the result of a first benchmarking run), which contributes to an essential objectification of the benchmarking process. Repetitions of the run with respect to possibly unexpected or undesirable results due to certain weightings are meaningless, since the weighting vector remains unchanged.
The Cognion Research Association applies this objective benchmarking or ranking approach to a wide range of different questions, ranging from the continually updated analysis of the competitiveness of 78 European regions as a location for industrial production and the analysis of the attractiveness of European metropolitan regions as a location for headquarter functions to the objectification of funding programs and job placement procedures.