Talk:Analytic hierarchy process

Strong POV comes through in text

I'm new here and I have to say that this article reads like it was written by one of Saaty's grad students. It's full of value-laden adjectives (positive for AHP, negative for everything else), and presents no account (balanced or otherwise) of the many criticisms of AHP. Given that rank reversal is considered a prima facie refutation of AHP by many economists, mathematicians, and operations researchers, any unbiased encycolpedia article should address it. DMTate (talk) 21:07, 7 July 2008 (UTC)

So others can better understand your concerns, please be more specific about the "value-laden adjectives (positive for AHP, negative for everything else)" that you see here. Lou Sander (talk) 17:00, 8 July 2008 (UTC)

DMTate and I must not be looking at the same article. I don't see any mention at all of "anything else," let alone any value-laden negative adjectives about it. There is a large criticisms section at the end of the article. Unfortunately it doesn't cite anything that can be accessed online. 74.1.175.146 (talk) 23:51, 9 July 2008 (UTC)

The Criticisms section references eight articles from the 1990s that contain a long debate about AHP. The section includes a single quotation from that debate (J.S. Dyer), with no other context, summary of the debate, etc. I propose to delete the quote, but retain the reference to the article from which it was taken. Skyrocket654 (talk) 15:13, 12 July 2008 (UTC)

I agree with Skyrocket654 that this quotation seems out of place in this part of the Criticisms section that seems to be more about laying the groundwork with a list of reference articles. Probably it should be removed, but the reference retained. MathDame (talk) 19:25, 22 July 2008 (UTC)

Done. I also propose to remove the Holder quote that follows the above, for the same reasons. It is unreferenced, but presumably is contained in one of the two citations that mention Holder. Skyrocket654 (talk) 16:41, 23 July 2008 (UTC)

BTW, I completely agree with what DMTate wrote above. It reads like a brochure for AHP; it always puts a positive spin on things.

E.g., "It is the nature of the AHP to promote focused discussions about difficult aspects of the decisions to which it is applied. Such discussions encourage the communication of differences, which in turn encourages cooperation, compromise, and agreement among the members of the group."

Even the tone of the first sentence of the criticism section biases the section. It suggests that there is broad consensus that AHP is sound, but somehow there are still criticisms. Burkander (talk) 20:51, 1 October 2008 (UTC)

1) DMTate talked about "value laden adjectives." Burkander "completely agrees" with him, and presents an illustration. The only adjectives in the illustration are "focused" and "difficult;" IMHO those words aren't very value-laden.

2) There IS broad consensus that AHP is sound. The reference more than "suggests" it, it states it outright. Further evidence of soundness is in the widespread use demonstrated in the Uses and Applications Section.

3) There are definitely criticisms of the method, mostly advanced in academic journals from many years ago. About a dozen of them are referenced in the Criticisms section. They don't lend themselves very well to exposition in an encyclopedia article, and neither do the answers or responses to them. Reference #21 deals with the whole matter at length; maybe it should be available online.

The reference is now available online, and is linked to in the article. Lou Sander (talk) 13:38, 16 October 2008 (UTC)

People who feel that the method is unsound are free not to use it, and some of them don't. But a lot of people DO use it, and they aren't unaware of the criticisms. Lou Sander (talk) 02:56, 2 October 2008 (UTC)

The article comes across as POV to me as well. The article fails to outlines the limitation of the model. The model has assumptions and those assumptions do not always hold. The article would feel alot more balanced if these assumptions were stated outright. 207.132.184.130 (talk) 19:23, 28 January 2009 (UTC)

Implementation / External links

I added a very short section mentioning the use of the AHP software and showing one of the gadgets. It references both of the companies that previously had entries in External links. I removed those links, which were both a bit promotional in their descriptions. I also removed an External link that duplicated one of the references in the body of the article. Good Cop (talk) 22:14, 27 July 2008 (UTC)

Rank reversal, etc.

In response to calls for more specific information, I added a large subsection about rank reversal. I also did some minor rejiggering of headings. Lou Sander (talk) 19:43, 12 October 2008 (UTC)

Somehow, a key article in Operations Research that I wrote with Saul Gass was not referenced. It explains rank reversal and the ideal mode in great detail. I've also added an example (to this article) of Presidential elections where rank reversals have been present in the recent past.

The reference to Arrow's Impossibility Theorem was very vague. In fact, Arrow's Impossibility Theorem is NOT impossible if ratio measure, such as that used in AHP is applied.

Schenkerman's article is based on a wrong assumption by Schenkerman. AHP will produce the exact results Shenkerman was looking for if applied correctly.

(Ernest Forman 23 November 2008). —Preceding unsigned comment added by EForman (talk • contribs) 14:51, 23 November 2008 (UTC)

The following material was inappropriately put in the article itself. I moved it here. Lou Sander (talk) 01:08, 1 December 2008 (UTC)

Comment: The above paragraph does not make sense to me. Your original table simply did not represent the number of available dresses of each type. As your example continues, the customer now takes into account the number of available dresses of each type (e.g. relative scarcity of each alternative), and so clearly the decision table should be modified to include this new criterion. With this new criterion in place, priorities can be re-assigned and alternatives (there will be no additional ones) re-evaluated. If now the ranking of alternatives is different, e.g. reverses, this is due to the additional criterion (relative scarcity). In fact, referring to the discovery of new dresses (of existing types) as "additional alternatives" or "copies" makes very little sense to me: a criterion was added, not an alternative. With respect to the subsequent example (the 2004 elections), there are no "copies" involved, rather you discuss the effect of a dominated alternative on the relative rankings of the dominating alternatives. 69.217.120.82 (talk)

• If the material was deemed inappropriate for the article itself, and hence moved here, it should have been addressed here (it has not been), or it should have been formatted/reworded to be suitable for the main article (it has not until now). I am adding this point of contention to the main article, in a more appropriate format. Please don't remove it again, without addressing the main point of contention / confusion. —Preceding unsigned comment added by 69.217.124.169 (talk) 03:07, 4 February 2009 (UTC)

Neutrality & Tone

The tone of this article reads a bit like a press release rather than a neutral article. In particular the section on criticisms is written in a manner that causes one to discount the criticisms. This section could also benefit from some mention of Arrow's Impossiblity Theorem192.237.29.125 (talk) 17:46, 18 November 2008 (UTC)

Strongly disagree. The criticisms are given their just due. Remove the neutrality tag, please. Good Cop (talk) 21:43, 18 November 2008 (UTC) —Preceding unsigned comment added by DCLawyer (talk • contribs)

The "unsigned" notation above refers to my moving this section to the end of the page. It formerly was out of place at the beginning. Sorry if anybody got confused. DCLawyer (talk) 14:09, 19 November 2008 (UTC)

Disagree. Remove the Neutrality tag. The anon calls the article "promotional" but provides no examples. The criticisms involving rank reversal are the major criticisms of the method. They are thoroughly discussed, with references. The "other criticisms" are old and/or have no discernable following. If there is any published material relating Arrow's theorem to the Analytical Hierarchy Process, it isn't cited. DCLawyer (talk) 14:09, 19 November 2008 (UTC)

This sentence for starters as the lead in to the section pushes people to doubt the criticisms:

Although the Analytic Hierarchy Process has been the subject of many research papers and the general consensus is that the technique is both technically valid and practically useful, there are critics of the method[4]

The sentence is from a cited source that is available online. It seems to sum up the state of affairs regarding criticism of AHP: the process is extremely widely used; its inventor keeps receiving important awards for inventing it; AND there are still occasional papers that advance one criticism or another. IMHO, those occasional papers are given undue weight in this section of the article. Skyrocket654 (talk) 00:01, 22 November 2008 (UTC)

In the early 1990s a series of debates between critics and proponents of AHP was published in Management Science.

This is so, and quite a number of them are cited as references. Given the age of this debate and the subsequent worldwide adoption of AHP by legions of serious users, there may be an undue weight problem here, too. But nobody is complaining about it. Skyrocket654 (talk) 00:01, 22 November 2008 (UTC)

This sentence as the lead in to the rank reversal section raises doubts about the criticism before raising explaining what it is:

"Most criticisms of AHP have involved a phenomenon called "rank reversal." Many people hear about rank reversal and assume that there is some sort of proven principle about it that needs to be upheld in making decisions. That unwarranted assumption has led to much misunderstanding of AHP and other decision making techniques."

There are plenty of doubts about the criticism, and they have been raised in numerous places. IMHO the subsequent long explanation of rank reversal, coupled with the online availability of a very long treatise on the subject, pretty much lay out the situation. Skyrocket654 (talk) 00:01, 22 November 2008 (UTC)

As for the Arrow's Impossiblity Theorem, most of the criticisms aren't easily availible publicly - the journarls are paid subscriptions. But the theorem itself as currently defined on wikipedia is a criticism:

"[Arrow's Impossiblity Theorem] demonstrates that no voting system can convert the ranked preferences of individuals into a community-wide ranking while also meeting a certain set of reasonable criteria with three or more discrete options to choose from" —Preceding unsigned comment added by 192.237.29.125 (talk) 17:26, 21 November 2008 (UTC)

If an editor wants to say that somebody's theorem involves criticism of AHP, it is their responsibility to find citations of that criticism in verifiable published sources. We can't just, on our own, cite somebody's theory and conclude that it must apply to AHP or anything else. Skyrocket654 (talk) 00:01, 22 November 2008 (UTC)

It is a proven fact, verifiable in any kitchen, that metal objects sink in water. This is NOT a legitimate criticism of metal ships. DCLawyer (talk) 06:07, 22 November 2008 (UTC)

Remove the neutrality tag. Except for rank reversal, the other criticisms are pretty obviously given undue weight. It would be good if all editors of the criticisms section would read WP:Undue and temper their enthusiasm accordingly. Skyrocket654 (talk) 14:23, 29 November 2008 (UTC)

Remove the neutrality tag. I agree that the other criticisms are given undue weight. MathDame (talk) 20:45, 2 December 2008 (UTC)

The biggest issue here is tone of the entire article. This reads like a press release not an encyclopedia entry.tone

Many editors seem to disagree with you. Please be specific about what you mean. Maybe use this page to quote something verbatim from the article and show us how you think it "reads like a press release."

With regards to the Criticisms section, it is pretty clear that criticisms from very old journal articles that failed to gain a following are described in inappropriate depth. The detailed quotation from "AHP critic Stan Schenkerman" is an example of this. His viewpoint is not significant or prominent. It is a ten-year-old view of one person, now deceased, yet it is described as "another problem {with AHP]." Not only that, but it concludes that "the decision maker relying on AHP or these variants can be seriously misled." WP:UNDUE says that such tiny minority views shouldn't even be in the article. IMHO, Schenkerman's point should perhaps be mentioned and cited, but that's about all. Skyrocket654 (talk) 16:57, 11 December 2008 (UTC)

I removed the tag, per the above discussion. DCLawyer (talk) 16:21, 13 December 2008 (UTC)

"SUBSTANTIATE"

I removed a bunch of these from the article. They don't belong there. There ARE ways to question material, but this isn't one of them. Skyrocket654 (talk) 17:01, 11 December 2008 (UTC)

Criticisms section

In keeping with the discussion in several sections above, I recast the criticisms section to fix the undue weight being given to minority viewpoints. The unsupported claim about Arrow's Impossibility Theorem was removed because it seems to be original research. The material in the 1997 and 2006 articles was reduced to a single (hopefully fair) sentence on each, and the citations were retained. The INFORMS Impact Prize was put into a footnote. Everything but Rank Reversal was collected and put at the beginning of the section. DCLawyer (talk) 16:02, 13 December 2008 (UTC)

I also removed the "unwarranted" from the "unwarranted assumption" about rank reversal. The word doesn't appear in the cited reference. DCLawyer (talk) 16:26, 13 December 2008 (UTC)

The article briefly mentioned a paper by Forman and Gass. I found a copy of it, and it has a lot of good stuff about the academic criticisms. I mentioned it more prominently in the Criticisms section, including a link to an online copy HERE. Lou Sander (talk) 04:11, 16 December 2008 (UTC)

I made a change to the first paragraph of this section. Someone had expressed concern that the previous lead sentence was based on a practical magazine for users, rather than on a peer-reviewed paper. It is now based on a peer-reviewed paper, with additional comment from the practical magazine.

IMHO the criticisms section is now thoroughly referenced to strong sources, and describes the situation completely and fairly: AHP's undeniably widespread use and thorough vetting are first briefly mentioned, with solid references. Then twelve specific critcal papers are mentioned and referenced, with descriptions of the main points of the most recent ones. (Though a case can be made that as tiny minority views the recent ones are given undue weight, nobody is complaining about it.) Then there is a long subsection explaining Rank Reversal, the phenomenon underlying almost all the criticisms. Finally, two full-text references that summarize and analyze the criticisms are provided -- one from a peer-reviewed journal (Forman and Gass), and one from a scholarly foundation text on AHP. Lou Sander (talk) 15:27, 19 December 2008 (UTC)

I agree that this is getting to be a nice piece of work. Good job, Lou Sander. MathDame (talk) 04:40, 20 December 2008 (UTC)

Support for Theory

An anonymous IP editor has split this off from the Criticisms section. There were no edit summaries. The user has a slightly questionable history. The material is about criticisms of AHP, and there doesn't seem to be any reason to split it off. I propose to put it back in the criticisms section. MathDame (talk) 17:05, 4 January 2009 (UTC)

I agree with MathDame. There's no sense in breaking off the new section. A week has passed with no comment. Put it as it was. Lou Sander (talk) 17:17, 11 January 2009 (UTC)

Support. Combine the two. 74.1.175.146 (talk) 01:29, 15 January 2009 (UTC)

Done.

MathDame (talk) 15:22, 15 January 2009 (UTC)

Summary section

I've redone the first part of the summary section without (I hope) changing its essence. The steps now correspond to those in the cited reference. I removed two claims that were challenged over ten months ago and well over a year ago and never responded to. Lou Sander (talk) 18:24, 11 January 2009 (UTC)

I renamed "Summary" as "Steps in using the process." The section has expanded greatly since the original heading was put in, and I think the new name is a more accurate description of what is actually there. I hope that others agree. Lou Sander (talk) 15:05, 22 January 2009 (UTC)

Decision situations

I added some material about the decision situations in which AHP is often used. It's more than just picking the best alternative. Lou Sander (talk) 17:42, 16 January 2009 (UTC)

Analytic Hierarchy Process/More Hierarchies

I have moved a page that was formerly at Analytic Hierarchy Process/More Hierarchies to Talk:Analytic Hierarchy Process/More Hierarchies. That page was apparently a draft page for proposed addditions to this page, and as such it was a subpage that should have been added to talk rather than article space. This note is jurt to help you find it. - Smerdis of Tlön (talk) 15:51, 26 January 2009 (UTC)

Thanks for handling it in an effective way. Lou Sander (talk) 17:54, 26 January 2009 (UTC)

Criticisms section

I removed a bunch of original research from this section. It was posted by an anonymous editor who seems to have made other inappropriate edits to the article over time. IMHO it doesn't seem to be vandalism, but it looks like somebody is trying to make a point but doesn't understand some basic things about how Wikipedia works. Lou Sander (talk) 00:16, 6 February 2009 (UTC)

I'm new to this but I had to respond to one of the claims in this article. At one point the article claims that the criticisms of AHP have been "settled in favor" of AHP and sites the INFORMS Impact Prize to Saaty as evidence of this. The Impact Prize is certainly no testament to whether a method works. The prize description clearly states "This is not a research award. The awards committee is not judging the quality of a body of work. Instead, emphasis will be placed on evaluating the breadth of the impact of an idea or body of research." See the INFORMS Impact Prize description here http://www.informs.org/index.php?c=116&kat=INFORMS+Impact+Prize

I was surprised to read that anyone would claim this is a "settled" issue. From what I see, INFORMS meetings are still heated arguments about this controversial method of reducing important decisions to a series of subjective pairwise comparisons. The reason why the prize is worded so carefully is because, I think, a prize that would effectively endorse the method as valid would never have gotten a majority vote in INFORMS. Of the 18 different prizes awarded by INFORMS, I would say the Impact prize is among the least like any validation of the theory. The John Von Neumann Theory Prize, on the other, might be used as evidence of general agreement and validation by the members of the validity of a method. If that prize were awarded, I would think that would suffice as evidence that the issue has been settled.

It seems clear that the awarding of this prize isn't any kind of evidence for the claim being made. The more recent citations listed, on the other hand, seem to make the point that the method still has serious issues and more continue to be found. Also, it appears to overempahsize rank reversal, which doesn't strike me as a key issue anymore now that there as at least some mathematical guard against it.

It seems that, at a minimum, any claim that the issue is "settled" should be removed since the citation provided clearly does not support that claim. Just my thoughts. DFLovett (talk) 22:10, 2 March 2009 (UTC)

We keep hearing from people who assert that AHP is "controversial," or "doesn't work," or is "dangerous," or "clearly" has some other problems, or isn't the widely-accepted tool that huge amounts of evidence show that it is. So far, these folks have never offered much to back up their assertions. On the other hand, the AHP tracks at important OR meetings keep growing and growing, the use of AHP is booming, the prizes keep coming, etc. It would be interesting to see a few coherent paragraphs on this page, backed by citations from the 21st century, about AHP's perceived drawbacks. What, specifically, are the problems? Why don't the naysayers seem to be gaining any traction? Or ARE they gaining traction and nobody has pointed it out here? Lou Sander (talk) 22:49, 2 March 2009 (UTC)

You mean evidence other than the published articles this very article cites? You asked for specific problems but the published articles already cited seem to do an adequate job in that regard, especially once the only "evidence" that they were settled (the Impact Award) is refuted. You asked for 21st century (as if the age of the flaw somehow addresses the flaw), but it looks like the Perez citation alone meets that requirement.

This wasn't the point of my comment but it would help if you showed the evidence that shows AHP measurably improves decisions in controlled experiments. There are many articles that are case studies about AHP beling used, but I find none that show that users of AHP beat some other method (or even intuition) in a repeated, well-defined decision making task. If you could find such an article - not just another case study where the users say they were satisfied - then it should certainly be listed and discussed here in at least as much detail as rank reversal. And I think it would satisfy many skeptics, myself included.

Regarding your point about the traction of skeptics, I would be just as curious to see a survey of decision analysis experts or INFORMS members as you. Otherwise, it would seem we are just winging it with personal opinions and anecdotes. All I can say is that in my circles AHP doesn't seem to be taken as a serious alternative to, say, probabilistic models of systems that use empirical measurements and are tested against repeated observation. Perhaps your circles are different. I might be more open to AHP than some of my peers because I think AHP might be relevant for perhaps a narrow set of problems that can only be satisfactorally described as a system of subjective pairwise comparisons. Again, that's why a survey would be great if anyone could find a citation. It would also help settle what appears to be the entirely subjective opinions about what is "undue weight" of various issues on this topic.

But this is not the main point I was making, either. The main point I was making was simply in regards to the use of the Impact Award as evidence that the controversy is settled. The language of the award would justify no such claim that this is some kind of endorsement of the method's validity. Regardless of anything else you agree to about the article or the merits of the method, it seems clear that the award does not "settle" the issue at all and should not be represented as such. Can we at least agree that the Impact Award is not a research award and explicitly is NOT a recognition of the quality of the work (as the award description states)?DFLovett (talk) 00:12, 3 March 2009 (UTC)

From the citation of the INFORMS Impact Award:

"The AHP has revolutionized how we resolve complex decision problems.... The AHP has been applied worldwide to help decision makers in every conceivable decision context across both the public and private sectors, with literally thousands of reported applications.

"The AHP is now an established methodology that is part of the OR curriculum for multi-criteria decision-making in business and engineering schools. Software for its implementation is readily available. The International Society on Multiple Criteria Decision Making awarded the MDCM Gold Medal to Professor Saaty in 2000 "...for the development of the Analytic Hierarchy Process (AHP) and the impact of his lifelong research contributions on several disciplines." Professor Saaty received the 2007 Akao Prize for excellence in Quality Function Deployment (QFD) from the QFD Institute.

"For his seminal work on the Analytic Hierarchy Process, and for its deployment and extraordinary impact, INFORMS is delighted to award the 2008 Impact Prize to Professor Thomas Saaty.

Your personal opinions are not germane. Don't be surprised if people don't respond to them. It doesn't do to merely besmirch the AHP or to present your own criticism of sources or arguments. It doesn't do to cherry-pick quotes from isolated papers (see WP:UNDUE). Try presenting a coherent description of published material showing that there are important criticisms and describing what they are. Good Cop (talk) 02:16, 3 March 2009 (UTC)

New example

I'm working on replacing the Jones car buying example with something a lot less complex. The Jones stuff will be retained, but moved out of the main article. Please be patient with the changes I'm making to set this up. Lou Sander (talk) 21:28, 24 February 2009 (UTC)