Written by Dawid Opydo
Analytic Hierarchy Process (AHP) without pairwise comparisons? Are you crazy? Is this even possible?
Well, pairwise comparisons are the core mechanism of AHP, the factor that makes it easy to use, and main reason why people love AHP. Yet people sometimes get so wrapped up in pairwise comparison and its elegance that they don’t realize that it sometimes gets in the way. Sometimes, scales are better. So we wrote this article to help you understand when to use each.
So, let’s get it out of the way. The General Rules for using pairwise comparisons, the rules that, if you follow them, you will get it right 95% of the time are…
Rule 1: Use pairwise comparisons to prioritize criteria.
Criteria are things that, by definition, can’t be measured on the same scale. You can’t come up with a scale that me asures risk and cost – they are just different.
But you still have to prioritize your criteria and this is where pairwise comparison comes in. Simple questions like, "Is risk or cost more important" are a great way to unpick the complexity of a multi-criterion decision and AHP gives us the tools to put the whole picture back together again.
Find out more: 3 Steps to Reduce the Number of Comparisons in AHP
Rule 2: Use scales to score alternatives.
Most people are usually fine with Rule 1. It makes sense, but they often get confused when looking at alternatives. They want to use pairwise comparisons to compare, say, different projects. This seems to make sense because the projects can be so different in nature as to be incomparable.
But here’s the thing. You are using criteria to assess the projects, and each criterion implies something you can measure, something you can directly compare between alternatives. For example, you might be trying to choose between a new car and a new pool, things that are totally different. Even so, you can still have measurable criteria like, "Cost" and "How frequently will I use it?" and "Will it impress my neighbors?"
Even though a car and a pool are totally different things, you can measure them on a specific scale for each criterion. You will use a car every day, all year round whereas you might only use a pool twice a year. So even though my alternatives are very different, they are easy to compare "like for like" for one criterion if you have a good scale. This helps you be objective and precise.
Oh, and if you can’t think of a good scale for your criterion, you might wonder if you’re at the bottom of the criterion tree, or is your criterion really made up of sub-criteria.
Like all good rules, of course, these rules should, sometimes, be broken. If you’re tempted to use pairwise comparisons to score alternatives, which is a very common urge, go through this checklist of questions – they will help you decide whether or not you can break the rules. In most cases, you’ll realize that scales will work better.
1. Are there fewer than 9 alternatives?
Don’t use pairwise comparison to score alternatives if there are more than about 9 alternatives. There are 3 main problems you will hit using pairwise comparisons to evaluate large number of alternatives:
It is time consuming. The more alternatives, the more comparisons. When you have multiple evaluators all of them need to provide judgments. Also, in group decisions, it will take much more time to build consensus. Pairwise comparisons for large set of alternatives might simply be too time consuming. But even if you did spend the time, there are still other problems…
Problems with consistency. With a large set of comparisons, it can be difficult to be consistent. With fewer alternatives, you are likely to be more consistent and, frankly, it can take a long time to identify and fix all the inconsistencies.
If you are inconsistent, TransparentChoice will identify that for you… but then what? Are you really going to spend all that time trying to tweak your answers to be consistent? Probably not. But even if you do complete comparisons with sufficient consistency, there is another problem…
Small differences in final scores. Pairwise comparison is way of working out how to "share out the pie". If A is more important than B, A will score higher than B. But if you have all the options, A through Z, then the pie has to be shared amongst 26 options. Those are small slices of pie and small changes in the pairwise comparison can lead to changes in ranking.
There are a couple of ways, in AHP theory land, to improve matters, but they do not fundamentally get around the pie problem. But scales do. Each alternative gets the score it deserves regardless of how many other alternatives there are.
NOTE: You will not have these problems with criteria. If you have more than 9 criteria, they will probably be organized as a hierarchy with some criteria sitting “under” others. Since you only compare criteria at the same level of the same branch, you rarely get a situation where you are trying to compare 9 criteria with each other.
2. Do you want to be able to score “0”?
If you want to be able to give something a score of zero, don’t use pairwise comparison. For example, if you have a scale for your Customer Support criterion, you might have 2=great, 1=limited support, 0=no support. There is no way, using pairwise comparison, to get that zero, so if you want to be able to use zero, use a scale.
The reason you can’t have zero is simple. Pairwise comparison asks, “Do you prefer option A or option B” and if you score a 3 in favor of option B, that means you like B thrice as much as A. This means that option B would carry 75% of the weight and A would carry 25%. If you score a 9 in favor of B (the strongest preference available) then B would carry 90% of the weight and A would carry just 10%... but 10% is not zero.
NOTE: You won’t have this problem with criteria. If criterion would have zero weight, well, you just delete it.
3. Will you want to add more alternatives later?
In some decisions, you might gradually add alternatives, or make a preliminary decision and then add other alternatives later. Project prioritization often works like that. If this is the case for your decision, use scales, not pairwise comparisons, to score your alternatives. Here’s why.
You need to compare a new alternative against all the old alternatives. If you use pairwise comparisons, as you add new alternatives, you need to “test” that new alternative against all the others. This isn’t just a logistical pain, it also means that your new judgments are being made at a different time and with different knowledge than the first set of judgments. This, in turn, means you’re looking through a different set of filters and will make different tradeoffs leading to less coherent conclusions.
You will affect ranking. Pairwise comparisons are like football leagues – if you add new teams, or remove existing teams – the whole situation in the league might change. The new team will influence the points collected by all the other teams in matches and… well, the table is likely to change.
Using scales eliminates both of these problems. Scales are “unambiguous” and don’t change over time. Similarly, if you use scales, the score of one alternative is not affected by other alternatives; you can add as many as you need.
NOTE: When you add a new criterion, it is intuitive that the final ranking will change.
4. Do your evaluators know enough?
Sometimes evaluators have deep knowledge on single alternative or subset of alternatives (projects, technologies, vendors, candidates) but not about all the alternatives that will be evaluated. It is hard to make pairwise comparisons in this case as you would need knowledge about all the alternatives.
In comparison, scales give you a clear yard-stick to use and different people can provide input for different alternatives (though it’s usually a good idea to have one person reviewing all of these inputs to ensure the scale is applied consistently).
NOTE: For criteria, you have to know about the different criteria and their impact on the business in order to prioritize. In fact, individuals usually don’t have perfect knowledge of all criteria, but a team will typically work this out at the consensus-building stage.
