[AM-03-013] Multi-Criteria Evaluation

Rinner, C. and Jankowski, P. (2024) Multi-criteria Evaluation. The Geographic Information Science & Technology Body of Knowledge (2024 Edition). John P. Wilson (Ed.).  DOI: 10.22224/gistbok/2024.1.21

1. Introduction and Motivation
2. Principles of Multi-Criteria Motivation
3. MCE Implementation in GIS
4. Theoretical Considerations in Geospatial MCE
5. Trends in Research, Development, and Applications of MCE in GIS

1. Introduction and Motivation

When we make decisions that are larger in scope and importance than ad-hoc, instinctive choices, we usually consider multiple criteria and conceive lists of pros and cons, whether on paper or in our mind. At its core, multi-criteria evaluation (MCE) is a systematic pro-con list with a calculated result. For example, a young family looking to buy one of two affordable houses in comparable locations might choose the house with two extras (e.g., a spare bedroom and a preferred heating system) over the house with only one extra (e.g., a garage). We may just count the number of pros against the number of cons for a simple yes/no choice, compare the number of pros between options, as in the example, or we may rate each option’s achievement on each criterion, weight some of the criteria higher than others, and combine the benefit (pro) and cost (con) criteria into a score representing the overall strength of each option.

Another way to look at MCE is through the lens of composite indices. Everyday applications of MCE in public life include rankings such as the most trendy neighborhoods, the top-10 cities to visit, the best universities for undergraduate studies, etc.. These rankings are usually based on indices comprising multiple indicators that capture various aspects of the alternatives being evaluated, such as cost of living, proximity of beaches or green spaces, crime rates, and more. More complex examples include metrics for urban quality of life such as the City of Toronto’s (2018) neighbourhood wellbeing indices or the U.S. Bureau of Labor Statistics’ (2023) Consumer Price Index (CPI).

The above examples also illustrate the implicit spatial nature of many evaluation or decision problems, and thus the relevance of MCE principles and techniques to students of GIS&T. Research into place- or area-based composite indices (Cutter 1985; Rinner and Pietropaolo 2021) also attests to this relevance. In this chapter, we briefly present principles of MCE followed by their implementation in GIS software, theoretical considerations around the geospatial aspects of MCE, and lastly, current trends in research, development, and applications of MCE in GIS.

2. Principles of Multi-Criteria Evaluation

We employ the term “evaluation” to include decision problems, i.e. the concrete choice between multiple alternatives, as well as more general situational assessments. The term MCE is often used interchangeably with “multi-criteria decision analysis” and similar terms, yet may imply a slightly broader scope. The constituting elements of an evaluation or decision problem include the set of alternatives, two or more criteria by which to rate the alternatives, and a combination rule (also called “aggregation function”) used to arrive at a summary score for each alternative. The criteria, which correspond to the “indicators” in the examples above, can be further classified into soft “factors” and hard “constraints” to highlight their differing roles in evaluation. High and low ratings on factors can be substituted for each other, while constraints function like database search and filter terms with a clear outcome of hit or miss. In the case of a homebuyer, the criteria might include general aspects like a home’s size, quality of heating/cooling, and car parking options. These could be measured by the number of bedrooms, presence and age of the HVAC system, or the availability of parking (street, driveway/front-yard parking, or garage), with higher ratings assigned to more desirable outcomes.

A number of specific MCE techniques represent different approaches to combining (aggregating) criterion weights with ratings, from weighted linear combination (also known as simple additive weighting) through its generalization, ordered weighted averaging (OWA), to the analytic hierarchy process (AHP) (Malczewski & Rinner 2015). AHP facilitates criterion weighting through pairwise comparison of relative criterion importance, while OWA adds a second set of weights that enable representing risk-taking or risk-averse decision strategies. In these techniques, the summary score of each alternative is typically calculated using a weighted sum. Other popular MCE methods include Ideal Point Analysis and methods based on outranking relationship (ELECTRE, PROMETHEE) (Joerin et al. 2001). Instead of the weighted sum, these techniques rely on the distance from a reference point or on determining the extent to which one alternative outranks another. Cost-benefit analysis, a widely used approach in business and policy decision-making, can be seen as a form of MCE. It similarly involves assessing and combining multiple attributes into a single summary metric to enable direct comparison of alternatives. Similarly, benefit-risk evaluation of drugs or non-pharmaceutical interventions is an essential step in clinical medicine, pharmacology, and public health.

Several fundamental issues with MCE can be illustrated using the AHP technique as an example. The AHP aims to simplify the criterion weighting process. The decision-maker determines the relative importance of each pair of criteria on a predefined scale, and percentage weights are automatically calculated from this input. A consistency measure signals whether the pairwise comparisons were logical. For example, the homebuyer might view home size as more important than car parking, and car parking as more important than heating/cooling, since this is the easiest feature to improve after a possible purchase. In this case, home size should also be deemed as (much) more important than heating/cooling; otherwise, an inconsistency in the buyer’s ratings would arise.

Possible logical inconsistencies in the decision-maker’s preferences can lead to the rank-reversal problem (Malczewski & Jankowski 2020) and are worsened if the range-sensitivity principle (Malczewski 2011) is violated. Rank reversal occurs when a change in the structure of the decision problem, such as the addition or removal of an alternative, results in an unwarranted switch of ranks between two other alternatives. The range-sensitivity principle stipulates that criterion weights should be based on the actual range of criterion values within the dataset, rather than on the theoretical importance of each criterion —a concept that may seem counter-intuitive. For example, cost is often considered the most important criterion, yet if all alternatives are within a narrow cost range, this criterion should receive a low weight or be omitted as in the homebuyer’s example, since small, meaningless cost differences could impact the result. Together, these issues raise the question of whether a single correct decision can ever truly be determined.

The nature of available input data presents another practical issue with MCE. The techniques mostly require numeric data yet many real-world phenomena are difficult to quantify and measure with known accuracy. The dataset representing the evaluation or decision problem is also expected to be comprehensive and, depending on the mathematical nature of the MCE combination rule (a WLC in the case of most AHP-based evaluations), the criteria ought to be measured independently, while real-world data often comprise unaccounted collinearities. For example, in a socio-economic analysis, variables representing income and education are often correlated at a population level. In the homebuyer example, home size and private parking may be positively correlated, and including them in the analysis might introduce redundancy, effectively inflating the final score of larger homes.

One of the most important and difficult-to-accept properties of MCE is its fundamental nature as a normative or optimization model rather than a descriptive or explanatory approach. In contrast to other statistical analysis techniques, where results emerge from the data, in MCE it is the analyst or decision-maker who determines the model parameters and thus the results. MCE techniques follow the normative approach illustrating what “ought to be” (Malczewski and Jankowski 2020) according to the values and preferences entered into the process. The bulk of the analyst’s work goes into developing the model parameters rather than into the interpretation of model results. Examples of the combination of normative and descriptive approaches include the use of collinearity diagnostics to avoid dependencies among manually selected criteria (e.g. Venkatesh & Parimalarenganayaki 2023) or principal components analysis to determine criteria and weights (e.g. Hazell and Rinner 2019).

3. MCE Implementation in GIS

The existence of two complementary geospatial data models, raster and vector, has implications for the implementation of MCE techniques in GIS. Decision alternatives associated with real-world locations are represented by raster cells or by points, lines, or polygons in the vector model. Accordingly, criteria are represented by multiple raster layers or by multiple fields in a feature attribute table. Typically, raster or field calculators in standard GIS software are used to weight and combine these criteria. In more complex problems, where a location alternative comprises multiple raster cells or vector features, alternatives and criteria are represented by aggregations of these elementary units of analysis, e.g. an optimal area for urban development being aggregated from individual cells or polygons as a result of the evaluation process.

An alternative computational approach involves a coupling of GIS with an external MCE module, requiring data exchange between GIS and MCE. While this approach offers more flexibility in using various MCE techniques not supported by GIS, it is more demanding in terms of development and user learning effort (i.e., a user must be familiar with both interfaces – GIS and MCE).

Clark Lab’s Idrisi GIS (now TerrSet) was one of the first software packages to include a dedicated MCE module operated through a wizard user interface in the raster model (Eastman et al. 1995). The MCE module, still available in the most recent version of TerrSet 2020, offers WLC and OWA techniques along with robust criteria standardization based on fuzzy sets transformation functions. Most recently, Esri introduced the Suitability Modeler within the Spatial Analyst extension of ArcGIS Pro 2.6, which as of this writing is offered as part of ArcGIS Pro 3.3. The tool supports the user in selecting criterion layers, rescaling their values to a common numeric scale, weighting and combining them. It also offers the capability of finding high suitability areas comprised of contiguous raster cells, which is often a challenge in implementing MCE in GIS. Typically, MCE solutions comprise non-contiguous individual mapping units, which fulfill the decision-making requirement of high suitability but violate the spatial requirement of contiguity. The vendor presents this tool as “an interactive, exploratory environment for creating and evaluating a suitability model” (Johnston 2020), in line with research on the value of geovisualization in support of data exploration and hypothesis generation (Andrienko et al. 2007). Both, TerrSet’s MCE module and the ArcGIS Suitability Modeler operate in the raster model, yet Malczewski (2006) showed that about half of published MCE studies at that time employed the vector model. The prototype of a vector-based, exploratory MCE tool, MCDA4ArcMap (Rinner and Voss 2013) garnered some interest in site selection (e.g., Abudeif et al. 2015; Rikalovic et al. 2015).

4. Theoretical Considerations in Geospatial MCE

The use of MCE in GIS requires consideration of issues of space and scale as well as spatio-temporal representation. For example, Rinner and Pietropaolo (2021) explored the impact of data standardization on the comparability of composite index scores calculated for multiple points in time. Using a simple case study with eight criteria derived from two Census data tables ten years apart, they found that an index based on change variables was more effective than indices based on separate or joint data preprocessing. Munn and Dragićević (2023) present another spatio-temporal MCE case study, employing joint suitability functions across four time points. Their findings reveal a general decline in the characteristics of residential dwelling units over this period.

The impact of spatial scale on MCE results is an example of the modifiable areal unit problem. To investigate the impact of scale on MCE results, Hazell and Rinner (2019) created biodiversity indices for nested spatial units at two different geographic scales and confirmed significant differences in model fit. Traditional a-spatial multi-criteria methods have been widely used for spatial problems, only implicitly considering spatial variability. These methods generally assume that model parameters and outcomes do not change across geographic space. However, there is a recent shift toward spatially explicit methods due to the limitations of conventional approaches.

Two approaches to spatially explicit MCE have emerged: geographically varying outcomes and local multi-criteria analysis. The former focuses on spatially explicit models where MCE’s structural components are derived from spatial relations and outcomes vary geographically. For example, Rinner & Heppleston (2006) proposed using spatial relations such as location, proximity, and direction as a basis for defining decision criteria. In a sample home buyer’s decision problem, they also introduced adjusted scores based on the influence of nearby decision alternatives. The local approach to MCE, in contrast, emphasizes local modeling in recognition of spatial heterogeneity of preferences, with spatial weights playing a key role. As an example of local multi-criteria analysis, Malczewski (2011) presented a local version of the WLC technique based upon a generalized range-sensitivity principle. It refers to the idea that the importance or weight of a criterion can vary depending on the range of values that criterion takes within a specific context. In other words, the decision-maker's preference with respect to a criterion may change based on how much variation exists in the criterion's values across different regions comprising the decision space.

5. Trends in Research, Development, and Applications of MCE in GIS

MCE models and methods integrate elements from rational decision-making and bounded/procedural rationality theories (Simon, 1957), which assume a rational decision-maker who understands the decision problem, identifies all options, knows their outcomes, and can rank them by evaluating trade-offs. These assumptions are mirrored in MCE concepts like preferences, trade-offs, objectives, and decision rules. Malczewski and Jankowski (2020) suggested that theoretical bases for MCE can be expanded by integrating behavioral concepts like bounded rationality, prospect theory, and regret theory. These theories, which blend normative methods with empirical research, aim to enhance the understanding of decision-making by showing that people evaluate gains and losses differently. For instance, prospect theory reveals that individuals typically require a higher gain to compensate for a potential loss, challenging traditional utility theory. This insight has not yet been fully explored in GIS-MCE methods, where current models do not account for the differing perceptions of gains and losses. One exception is presented by Shen et al. (2021), who implemented the Logic Scoring of Preference technique in a GIS and tested it in an urban development case study. Future research in MCE could focus on validating prospect theory in spatially explicit decision contexts and utilizing large datasets on individual choices to estimate gain and loss utility functions, particularly for location-based decisions like selecting points-of-interest (POIs) (Malczewski and Jankowski, 2020).

Another recent trend in developing MCE methods and their applications for spatial problems involves a data-driven approach to deriving model parameter values. Multi-criteria models typically have used knowledge-driven estimates of parameters, relying on agents' expertise and judgments. However, the knowledge-driven approach can lead to inconsistent results as agents may struggle to clearly express their preferences. Parameters like criterion weights and value functions can also be derived from data. An example of this approach is the use of Machine Learning techniques such as Random Forest, Extreme Gradient Boosting, and Support Vector Machine to support criterion weighting in GIS-MCE (Zhao et al., 2024).

Other recent advancements in MCE methods have focused on uncertainty inherent in parameters such as criterion weights and scores, and its impact on the reliability of model outcomes (Ligmann-Zielinska et al., 2024). To address this, an integrated approach combining uncertainty and sensitivity analysis, particularly through global sensitivity analysis methods, has been proposed to enhance the assessment of MCE results' reliability. Emerging practical applications of geospatial MCE will benefit greatly from these improvements. They include societally relevant applications such as three-dimensional residential real estate assessment (Munn and Dragićević 2023), renewable energy planning (Tsakalerou et al. 2022; Shi et al. 2024), and geomarketing for post-secondary education (Ernawati et al. 2021). The students recruited using these methods may very well be the ones who will further develop MCE within GIST!