Jump to content

Delphi method: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
m link [eE]conomic trend
→Applications: - Added section on use in public health
Line 64: Line 64:
* the use of two or more panels representing different groups (such as policy-makers, experts, citizens), which the administrator can give tasks reflecting their diverse roles and expertise, and make them to interact within ad hoc communication structures. For example, the ''policy community'' members (policy-makers and experts) may interact as part of the ''main conference'' panel, while they receive inputs from a ''virtual community'' (citizens, associations etc.) involved in a ''side conference''. These web-based variable communication structures, which he calls ''Hyperdelphi'' (HD), are designed to make Delphi conferences "more fluid and adapted to the hypertextual and interactive nature of digital communication".
* the use of two or more panels representing different groups (such as policy-makers, experts, citizens), which the administrator can give tasks reflecting their diverse roles and expertise, and make them to interact within ad hoc communication structures. For example, the ''policy community'' members (policy-makers and experts) may interact as part of the ''main conference'' panel, while they receive inputs from a ''virtual community'' (citizens, associations etc.) involved in a ''side conference''. These web-based variable communication structures, which he calls ''Hyperdelphi'' (HD), are designed to make Delphi conferences "more fluid and adapted to the hypertextual and interactive nature of digital communication".
One successful example of a (partially) web-based policy Delphi is the five-round Delphi exercise (with 1,454 contributions) for the creation of the [[eLAC Action Plans]] in Latin America. It is believed to be the most extensive online participatory policy-making foresight exercise in the history of intergovernmental processes in the developing world at this time.<ref name="HilbertMilesOthmer2009"/> In addition to the specific policy guidance provided, the authors list the following lessons learned include "(1) the potential of Policy Delphi methods to introduce transparency and accountability into public decision-making, especially in developing countries; (2) the utility of foresight exercises to foster multi-agency networking in the development community; (3) the usefulness of embedding foresight exercises into established mechanisms of representative democracy and international multilateralism, such as the United Nations; (4) the potential of online tools to facilitate participation in resource-scarce developing countries; and (5) the resource-efficiency stemming from the scale of international foresight exercises, and therefore its adequacy for resource-scarce regions."<ref name="HilbertMilesOthmer2009"/>
One successful example of a (partially) web-based policy Delphi is the five-round Delphi exercise (with 1,454 contributions) for the creation of the [[eLAC Action Plans]] in Latin America. It is believed to be the most extensive online participatory policy-making foresight exercise in the history of intergovernmental processes in the developing world at this time.<ref name="HilbertMilesOthmer2009"/> In addition to the specific policy guidance provided, the authors list the following lessons learned include "(1) the potential of Policy Delphi methods to introduce transparency and accountability into public decision-making, especially in developing countries; (2) the utility of foresight exercises to foster multi-agency networking in the development community; (3) the usefulness of embedding foresight exercises into established mechanisms of representative democracy and international multilateralism, such as the United Nations; (4) the potential of online tools to facilitate participation in resource-scarce developing countries; and (5) the resource-efficiency stemming from the scale of international foresight exercises, and therefore its adequacy for resource-scarce regions."<ref name="HilbertMilesOthmer2009"/>

=== Use in public health ===
The Delphi technique has been applied in developing consensus on public health research and policy recommendations.

An example of rigorous application of the Delphi methodology involved core group convenings, three rounds of data collection and revision, and ranking of a series of statements and recommendations from a  multi-national panel for ending the COVID-19 public health threat.<ref name=":0">{{Cite journal |last=Lazarus |first=Jeffrey V. |last2=Romero |first2=Diana |last3=Kopka |first3=Christopher J. |last4=Karim |first4=Salim Abdool |last5=Abu-Raddad |first5=Laith J. |last6=Almeida |first6=Gisele |last7=Baptista-Leite |first7=Ricardo |last8=Barocas |first8=Joshua A. |last9=Barreto |first9=Mauricio L. |last10=Bar-Yam |first10=Yaneer |last11=Bassat |first11=Quique |last12=Batista |first12=Carolina |last13=Bazilian |first13=Morgan |last14=Chiou |first14=Shu-Ti |last15=del Rio |first15=Carlos |date=2022-11 |title=A multinational Delphi consensus to end the COVID-19 public health threat |url=https://www.nature.com/articles/s41586-022-05398-2 |journal=Nature |language=en |volume=611 |issue=7935 |pages=332–345 |doi=10.1038/s41586-022-05398-2 |issn=1476-4687 |pmc=PMC9646517 |pmid=36329272}}</ref> The panel was nonpartisan, voluntary, multidisciplinary with 386 panelists from 112 countries and territories participating. <ref name=":0" /> In addition to the specific public health statements and recommendations provided in the study, the authors note the following about the application of the Delphi technique: “Although our study reflects relatively few areas of disagreement, we believe that highlighting the key areas of disagreement may be instructive for decision-makers in their own prioritization processes addressing the COVID-19 pandemic.” <ref name=":0" />

The Delphi technique has also been applied to several other areas of public health research and risk including, but not limited to, non-alcoholic fatty liver disease<ref>{{Cite journal |last=Lazarus |first=Jeffrey V. |last2=Mark |first2=Henry E. |last3=Anstee |first3=Quentin M. |last4=Arab |first4=Juan Pablo |last5=Batterham |first5=Rachel L. |last6=Castera |first6=Laurent |last7=Cortez-Pinto |first7=Helena |last8=Crespo |first8=Javier |last9=Cusi |first9=Kenneth |last10=Dirac |first10=M. Ashworth |last11=Francque |first11=Sven |last12=George |first12=Jacob |last13=Hagström |first13=Hannes |last14=Huang |first14=Terry T.-K. |last15=Ismail |first15=Mona H. |date=2022-01 |title=Advancing the global public health agenda for NAFLD: a consensus statement |url=https://www.nature.com/articles/s41575-021-00523-4 |journal=Nature Reviews Gastroenterology & Hepatology |language=en |volume=19 |issue=1 |pages=60–78 |doi=10.1038/s41575-021-00523-4 |issn=1759-5053}}</ref>, iodine deficiency disorders<ref>{{Cite journal |last=Schaffner |first=Monika |last2=Rochau |first2=Ursula |last3=Stojkov |first3=Igor |last4=Qerimi Rushaj |first4=Vjollca |last5=Völzke |first5=Henry |last6=Marckmann |first6=Georg |last7=Lazarus |first7=John H. |last8=Oberaigner |first8=Wilhelm |last9=Siebert |first9=Uwe |date=2021-04-01 |title=Barriers Against Prevention Programs for Iodine Deficiency Disorders in Europe: A Delphi Study |url=https://www.liebertpub.com/doi/10.1089/thy.2020.0065 |journal=Thyroid |volume=31 |issue=4 |pages=649–657 |doi=10.1089/thy.2020.0065 |issn=1050-7256}}</ref>, building responsive health systems for communities affected by migration<ref>{{Cite journal |last=Pottie |first=Kevin |last2=Hui |first2=Charles |last3=Rahman |first3=Prinon |last4=Ingleby |first4=David |last5=Akl |first5=Elie A. |last6=Russell |first6=Grant |last7=Ling |first7=Li |last8=Wickramage |first8=Kolitha |last9=Mosca |first9=Davide |last10=Brindis |first10=Claire D. |date=2017-02 |title=Building Responsive Health Systems to Help Communities Affected by Migration: An International Delphi Consensus |url=https://www.mdpi.com/1660-4601/14/2/144 |journal=International Journal of Environmental Research and Public Health |language=en |volume=14 |issue=2 |pages=144 |doi=10.3390/ijerph14020144 |issn=1660-4601 |pmc=PMC5334698 |pmid=28165380}}</ref>, and the role of health systems in advancing well-being for those living with HIV<ref>{{Cite journal |last=Lazarus |first=Jeffrey V. |last2=Safreed-Harmon |first2=Kelly |last3=Kamarulzaman |first3=Adeeba |last4=Anderson |first4=Jane |last5=Leite |first5=Ricardo Baptista |last6=Behrens |first6=Georg |last7=Bekker |first7=Linda-Gail |last8=Bhagani |first8=Sanjay |last9=Brown |first9=Darren |last10=Brown |first10=Graham |last11=Buchbinder |first11=Susan |last12=Caceres |first12=Carlos |last13=Cahn |first13=Pedro E. |last14=Carrieri |first14=Patrizia |last15=Caswell |first15=Georgina |date=2021-07-16 |title=Consensus statement on the role of health systems in advancing the long-term well-being of people living with HIV |url=https://www.nature.com/articles/s41467-021-24673-w |journal=Nature Communications |language=en |volume=12 |issue=1 |pages=4450 |doi=10.1038/s41467-021-24673-w |issn=2041-1723 |pmc=PMC8285468 |pmid=34272399}}</ref>.


===Use in reporting guidelines===
===Use in reporting guidelines===

Revision as of 00:09, 21 December 2022

Futures studies
Concepts
Techniques
Technology assessment and forecasting
Related topics

The Delphi method or Delphi technique (/ˈdɛlfaɪ/ DEL-fy; also known as Estimate-Talk-Estimate or ETE) is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts.[1][2][3][4] The technique can also be adapted for use in face-to-face meetings, and is then called mini-Delphi. Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach, prediction markets.[5]

Delphi is based on the principle that forecasts (or decisions) from a structured group of individuals are more accurate than those from unstructured groups.[6] The experts answer questionnaires in two or more rounds. After each round, a facilitator or change agent[7] provides an anonymised summary of the experts' forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel. It is believed that during this process the range of the answers will decrease and the group will converge towards the "correct" answer. Finally, the process is stopped after a predefined stop criterion (e.g., number of rounds, achievement of consensus, stability of results), and the mean or median scores of the final rounds determine the results.[8]

Special attention has to be paid to the formulation of the Delphi theses and the definition and selection of the experts in order to avoid methodological weaknesses that severely threaten the validity and reliability of the results.[9][10]

History

The name Delphi derives from the Oracle of Delphi, although the authors of the method were unhappy with the oracular connotation of the name, "smacking a little of the occult".[11] The Delphi method assumes that group judgments are more valid than individual judgments.

The Delphi method was developed at the beginning of the Cold War to forecast the impact of technology on warfare.[12] In 1944, General Henry H. Arnold ordered the creation of the report for the U.S. Army Air Corps on the future technological capabilities that might be used by the military.

Different approaches were tried, but the shortcomings of traditional forecasting methods, such as theoretical approach, quantitative models or trend extrapolation, quickly became apparent in areas where precise scientific laws have not been established yet. To combat these shortcomings, the Delphi method was developed by Project RAND during the 1950-1960s (1959) by Olaf Helmer, Norman Dalkey, and Nicholas Rescher.[13] It has been used ever since, together with various modifications and reformulations, such as the Imen-Delphi procedure.[14]

Experts were asked to give their opinion on the probability, frequency, and intensity of possible enemy attacks. Other experts could anonymously give feedback. This process was repeated several times until a consensus emerged.

A research protocol explaining the rigorous approach to applying the Delphi method were originally published in the BMJ Open in 2015.[15] This research protocol is generally used and cited now by any research applying the Delphi method as it is the first time a clear protocol has been described for the application of the methodology in practice.

In 2021, a cross-disciplinary study by Beiderbeck et al. focused on new directions and advancements of the Delphi method, including Real-time Delphi formats. The authors provide a methodological toolbox for designing Delphi surveys including among others sentiment analyses of the field of psychology.[16]

Key characteristics

The Delphi method communication structure

The following key characteristics of the Delphi method help the participants to focus on the issues at hand and separate Delphi from other methodologies: in this technique a panel of experts is drawn from both inside and outside the organisation. The panel consists of experts having knowledge of the area requiring decision making. Each expert is asked to make anonymous predictions.

Anonymity of the participants

Usually all participants remain anonymous. Their identity is not revealed, even after the completion of the final report. This prevents the authority, personality, or reputation of some participants from dominating others in the process. Arguably, it also frees participants (to some extent) from their personal biases, minimizes the "bandwagon effect" or "halo effect", allows free expression of opinions, encourages open critique, and facilitates admission of errors when revising earlier judgments.

Structuring of information flow

The initial contributions from the experts are collected in the form of answers to questionnaires and their comments to these answers. The panel director controls the interactions among the participants by processing the information and filtering out irrelevant content. This avoids the negative effects of face-to-face panel discussions and solves the usual problems of group dynamics.

Regular feedback

The Delphi method allows participants to comment on the responses of others, the progress of the panel as a whole, and to revise their own forecasts and opinions in real time.

Role of the facilitator

The person coordinating the Delphi method is usually known as a facilitator or Leader, and facilitates the responses of their panel of experts, who are selected for a reason, usually that they hold knowledge on an opinion or view. The facilitator sends out questionnaires, surveys etc. and if the panel of experts accept, they follow instructions and present their views. Responses are collected and analyzed, then common and conflicting viewpoints are identified. If consensus is not reached, the process continues through thesis and antithesis, to gradually work towards synthesis, and building consensus.

During the past decades, facilitators have used many different measures and thresholds to measure the degree of consensus or dissent. A comprehensive literature review and summary is compiled in an article by von der Gracht.[17]

Applications

Use in forecasting

First applications of the Delphi method were in the field of science and technology forecasting. The objective of the method was to combine expert opinions on likelihood and expected development time, of the particular technology, in a single indicator. One of the first such reports, prepared in 1964 by Gordon and Helmer, assessed the direction of long-term trends in science and technology development, covering such topics as scientific breakthroughs, population control, automation, space progress, war prevention and weapon systems. Other forecasts of technology were dealing with vehicle-highway systems, industrial robots, intelligent internet, broadband connections, and technology in education.

Later the Delphi method was applied in other places, especially those related to public policy issues, such as economic trends, health and education. It was also applied successfully and with high accuracy in business forecasting. For example, in one case reported by Basu and Schroeder (1977),[18] the Delphi method predicted the sales of a new product during the first two years with inaccuracy of 3–4% compared with actual sales. Quantitative methods produced errors of 10–15%, and traditional unstructured forecast methods had errors of about 20%. (This is only one example; the overall accuracy of the technique is mixed.)

The Delphi method has also been used as a tool to implement multi-stakeholder approaches for participative policy-making in developing countries. The governments of Latin America and the Caribbean have successfully used the Delphi method as an open-ended public-private sector approach to identify the most urgent challenges for their regional ICT-for-development eLAC Action Plans.[19] As a result, governments have widely acknowledged the value of collective intelligence from civil society, academic and private sector participants of the Delphi, especially in a field of rapid change, such as technology policies.

Use in patent participation identification

In the early 1980s Jackie Awerman of Jackie Awerman Associates, Inc. designed a modified Delphi method for identifying the roles of various contributors to the creation of a patent-eligible product. (Epsilon Corporation, Chemical Vapor Deposition Reactor) The results were then used by patent attorneys to determine bonus distribution percentage to the general satisfaction of all team members.[citation needed]

Use in policy-making

From the 1970s, the use of the Delphi technique in public policy-making introduces a number of methodological innovations. In particular:

  • the need to examine several types of items (not only forecasting items but, typically, issue items, goal items, and option items) leads to introducing different evaluation scales which are not used in the standard Delphi. These often include desirability, feasibility (technical and political) and probability, which the analysts can use to outline different scenarios: the desired scenario (from desirability), the potential scenario (from feasibility) and the expected scenario (from probability);
  • the complexity of issues posed in public policy-making tends to increased weighting of panelists’ arguments, such often solicited pro and con each item along with new items for panel consideration;
  • likewise, methods measuring panel evaluations tend to increased sophistication such as multi-dimensional scaling.

Further innovations come from the use of computer-based (and later web-based) Delphi conferences. According to Turoff and Hiltz,[20] in computer-based Delphis:

  • the iteration structure used in the paper Delphis, which is divided into three or more discrete rounds, can be replaced by a process of continuous (roundless) interaction, enabling panelists to change their evaluations at any time;
  • the statistical group response can be updated in real-time, and shown whenever a panelist provides a new evaluation.

According to Bolognini,[21] web-based Delphis offer two further possibilities, relevant in the context of interactive policy-making and e-democracy. These are:

A web-based communication structure (Hyperdelphi)[21]
  • the involvement of a large number of participants,
  • the use of two or more panels representing different groups (such as policy-makers, experts, citizens), which the administrator can give tasks reflecting their diverse roles and expertise, and make them to interact within ad hoc communication structures. For example, the policy community members (policy-makers and experts) may interact as part of the main conference panel, while they receive inputs from a virtual community (citizens, associations etc.) involved in a side conference. These web-based variable communication structures, which he calls Hyperdelphi (HD), are designed to make Delphi conferences "more fluid and adapted to the hypertextual and interactive nature of digital communication".

One successful example of a (partially) web-based policy Delphi is the five-round Delphi exercise (with 1,454 contributions) for the creation of the eLAC Action Plans in Latin America. It is believed to be the most extensive online participatory policy-making foresight exercise in the history of intergovernmental processes in the developing world at this time.[19] In addition to the specific policy guidance provided, the authors list the following lessons learned include "(1) the potential of Policy Delphi methods to introduce transparency and accountability into public decision-making, especially in developing countries; (2) the utility of foresight exercises to foster multi-agency networking in the development community; (3) the usefulness of embedding foresight exercises into established mechanisms of representative democracy and international multilateralism, such as the United Nations; (4) the potential of online tools to facilitate participation in resource-scarce developing countries; and (5) the resource-efficiency stemming from the scale of international foresight exercises, and therefore its adequacy for resource-scarce regions."[19]

Use in public health

The Delphi technique has been applied in developing consensus on public health research and policy recommendations.

An example of rigorous application of the Delphi methodology involved core group convenings, three rounds of data collection and revision, and ranking of a series of statements and recommendations from a  multi-national panel for ending the COVID-19 public health threat.[22] The panel was nonpartisan, voluntary, multidisciplinary with 386 panelists from 112 countries and territories participating. [22] In addition to the specific public health statements and recommendations provided in the study, the authors note the following about the application of the Delphi technique: “Although our study reflects relatively few areas of disagreement, we believe that highlighting the key areas of disagreement may be instructive for decision-makers in their own prioritization processes addressing the COVID-19 pandemic.” [22]

The Delphi technique has also been applied to several other areas of public health research and risk including, but not limited to, non-alcoholic fatty liver disease[23], iodine deficiency disorders[24], building responsive health systems for communities affected by migration[25], and the role of health systems in advancing well-being for those living with HIV[26].

Use in reporting guidelines

The guidance for developing reporting guidelines in 2010[27] recommended using the Delphi method for developing reporting guidelines. However, less than 30% of the reporting guidelines were developed with the Delphi method according to the systematic reviews in 2015[28] and 2020.[29] The Delphi approach has been successfully employed to improve agreement between experts in iterative rating exercises on clinical test results using inter-rater kappa statistics as an outcome measure.[30]

Online Delphi systems

Main article: Real-time Delphi

A number of Delphi forecasts are conducted using web sites that allow the process to be conducted in real-time. For instance, the TechCast Project uses a panel of 100 experts worldwide to forecast breakthroughs in all fields of science and technology. Another example is the Horizon Project, where educational futurists collaborate online using the Delphi method to come up with the technological advancements to look out for in education for the next few years.

Variations

Traditionally the Delphi method has aimed at a consensus of the most probable future by iteration. Other versions, such as the Policy Delphi,[31][32] is instead a decision support method aiming at structuring and discussing the diverse views of the preferred future. In Europe, more recent web-based experiments have used the Delphi method as a communication technique for interactive decision-making and e-democracy.[33] The Argument Delphi, developed by Osmo Kuusi, focuses on ongoing discussion and finding relevant arguments rather than focusing on the output. The Disaggregative Policy Delphi, developed by Petri Tapio, uses cluster analysis as a systematic tool to construct various scenarios of the future in the latest Delphi round.[34] The respondent's view on the probable and the preferable future are dealt with as separate cases. The computerization of Argument Delphi is relatively difficult because of several problems like argument resolution, argument aggregation and argument evaluation. The computerization of Argument Delphi, developed by Sadi Evren Seker, proposes solutions to such problems.[35]

Accuracy

Today the Delphi method is a widely accepted forecasting tool and has been used successfully for thousands of studies in areas varying from technology forecasting to drug abuse.[36] Overall the track record of the Delphi method is mixed. There have been many cases when the method produced poor results. Still, some authors attribute this to poor application of the method and not to the weaknesses of the method itself.[citation needed] It must also be realized that in areas such as science and technology forecasting, the degree of uncertainty is so great that exact and always correct predictions are impossible, so a high degree of error is to be expected.

A particular weakness of the Delphi method is that future developments are not always predicted correctly by consensus of experts. This shortcoming concerning the issue of ignorance is important. If panelists are misinformed about a topic, the use of Delphi may only add confidence to their ignorance.[37]

One of the initial problems of the method was its inability to make complex forecasts with multiple factors. Potential future outcomes were usually considered as if they had no effect on each other. Later on, several extensions to the Delphi method were developed to address this problem, such as cross impact analysis, that takes into consideration the possibility that the occurrence of one event may change probabilities of other events covered in the survey. Still the Delphi method can be used most successfully in forecasting single scalar indicators.

Delphi vs. prediction markets

Delphi has characteristics similar to prediction markets as both are structured approaches that aggregate diverse opinions from groups. Yet, there are differences that may be decisive for their relative applicability for different problems.[5]

Some advantages of prediction markets derive from the possibility to provide incentives for participation.

  1. They can motivate people to participate over a long period of time and to reveal their true beliefs.
  2. They aggregate information automatically and instantly incorporate new information in the forecast.
  3. Participants do not have to be selected and recruited manually by a facilitator. They themselves decide whether to participate if they think their private information is not yet incorporated in the forecast.

Delphi seems to have these advantages over prediction markets:

  1. Participants reveal their reasoning
  2. It is easier to maintain confidentiality
  3. Potentially quicker forecasts if experts are readily available.
  4. Delphi is applicable in situations where the bets involved might affect the value of the currency used in bets (e.g. a bet on the collapse of the dollar made in dollars might have distorted odds).

More recent research has also focused on combining both, the Delphi technique and prediction markets. More specifically, in a research study at Deutsche Börse elements of the Delphi method had been integrated into a prediction market.[38]

See also

References

  1. ^ Dalkey, Norman; Helmer, Olaf (1963). "An Experimental Application of the Delphi Method to the use of experts". Management Science. 9 (3): 458–467. doi:10.1287/mnsc.9.3.458. hdl:2027/inu.30000029301680.
  2. ^ Bernice B. Brown (1968). "Delphi Process: A Methodology Used for the Elicitation of Opinions of Experts.": An earlier paper published by RAND (Document No: P-3925, 1968, 15 pages)
  3. ^ Sackman, H. (1974), "Delphi Assessment: Expert Opinion, Forecasting and Group Process", R-1283-PR, April 1974. Brown, Thomas, "An Experiment in Probabilistic Forecasting", R-944-ARPA, 1972
  4. ^ Linstone, Harold A.; Turoff, Murray, eds. (1975), The Delphi Method: Techniques and Applications, Reading, Mass.: Addison-Wesley, ISBN 978-0-201-04294-8, archived from the original on 2008-05-20
  5. ^ a b Green, Armstrong, and Graefe (2007): Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared. Foresight: The International Journal of Applied Forecasting (Fall 2007). PDF format[1]
  6. ^ Rowe and Wright (2001): Expert Opinions in Forecasting. Role of the Delphi Technique. In: Armstrong (Ed.): Principles of Forecasting: A Handbook of Researchers and Practitioners, Boston: Kluwer Academic Publishers.
  7. ^ McLaughlin, Milbrey W. (1990). "The Rand Change Agent Study Revisited: Macro Perspectives and Micro Realities". Educational Researcher. 19 (9): 11–16. doi:10.2307/1176973. ISSN 0013-189X.
  8. ^ Rowe and Wright (1999): The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting, Volume 15, Issue 4, October 1999.
  9. ^ Markmann, C. et al. (2020) Improving the question formulation in Delphi‐like surveys: Analysis of the effects of abstract language and amount of information on response behavior. In: Futures & Foresight Science. e56, doi:10.1002/ffo2.56.
  10. ^ Mauksch et al. (2020) Who is an expert for foresight? A review of identification methods. In: Technological Forecasting and Social Change. Vol. 154, 119982, doi:10.1016/j.techfore.2020.119982.
  11. ^ Adler, Michael & Erio Ziglio (1996) Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, (Jessica Kingsley Publishers, 1996). ([2])
  12. ^ "JVTE v15n2: The Modified Delphi Technique - A Rotational Modification," Journal of Vocational and Technical Education, Volume 15 Number 2, Spring 1999, web: VT-edu-JVTE-v15n2: of Delphi Technique developed by Olaf Helmer and Norman Dalkey.
  13. ^ Rescher(1998): Predicting the Future, (Albany, NY: State University of New York Press, 1998). ([3], [4], [5])
  14. ^ Passig, David (1997) "Imen Delphi: A Delphi Variant Procedure for Emergence". Human Organization: Journal of the Society for Applied Anthropology, Southern Methodist University. Dallas, TX. Spring, 56(1), 53-63
  15. ^ Etherton-Beer, Christopher; McLachlan, Andrew; Clifford, Rhonda; Potter, Kathleen; Page, Amy (2015-08-01). "Prescribing for Australians living with dementia: study protocol using the Delphi technique". BMJ Open. 5 (8): e008048. doi:10.1136/bmjopen-2015-008048. ISSN 2044-6055. PMC 4538244. PMID 26264272.
  16. ^ Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. In: MethodsX. Volume 8, 2021, 101401, doi:10.1016/j.mex.2021.101401.
  17. ^ von der Gracht, Heiko (2012). "Consensus Measurement in Delphi Studies - Review and Implications for Future Quality Assurance". Technological Forecasting and Social Change. 79 (8): 1525–1536. doi:10.1016/j.techfore.2012.04.013.
  18. ^ Basu, Shankar; Roger G. Schroeder (May 1977). "Incorporating Judgments in Sales Forecasts: Application of the Delphi Method at American Hoist & Derrick". Interfaces. 7 (3): 18–27. doi:10.1287/inte.7.3.18.
  19. ^ a b c Hilbert, Martin; Miles, Ian; Othmer, Julia (2009). "Foresight tools for participative policy-making in inter-governmental processes in developing countries: Lessons learned from the eLAC Policy Priorities Delphi" (PDF). Technological Forecasting and Social Change. 15 (2): 880–896. doi:10.1016/j.techfore.2009.01.001.
  20. ^ Murray Turoff, Starr Roxanne Hiltz, "Computer-based Delphi processes", in Michael Adler, Erio Ziglio (eds.), Gazing Into the Oracle 1996, op. cit.
  21. ^ a b Bolognini, Maurizio (2001), Democrazia elettronica. Metodo Delphi e politiche pubbliche (Electronic Democracy. Delphi Method and Public Policy-Making) (in Italian), Rome: Carocci Editore, ISBN 978-88-430-2035-5. A summary is also in Glenn, Jerome C.; Gordon, Theodore J., eds. (2009), "23", The Millennium Project. Futures Research Methodology, New York: Amer Council for the United Nations, ISBN 978-0981894119, chap. 23.
  22. ^ a b c Lazarus, Jeffrey V.; Romero, Diana; Kopka, Christopher J.; Karim, Salim Abdool; Abu-Raddad, Laith J.; Almeida, Gisele; Baptista-Leite, Ricardo; Barocas, Joshua A.; Barreto, Mauricio L.; Bar-Yam, Yaneer; Bassat, Quique; Batista, Carolina; Bazilian, Morgan; Chiou, Shu-Ti; del Rio, Carlos (2022-11). "A multinational Delphi consensus to end the COVID-19 public health threat". Nature. 611 (7935): 332–345. doi:10.1038/s41586-022-05398-2. ISSN 1476-4687. PMC 9646517. PMID 36329272. {{cite journal}}: Check date values in: |date= (help)CS1 maint: PMC format (link)
  23. ^ Lazarus, Jeffrey V.; Mark, Henry E.; Anstee, Quentin M.; Arab, Juan Pablo; Batterham, Rachel L.; Castera, Laurent; Cortez-Pinto, Helena; Crespo, Javier; Cusi, Kenneth; Dirac, M. Ashworth; Francque, Sven; George, Jacob; Hagström, Hannes; Huang, Terry T.-K.; Ismail, Mona H. (2022-01). "Advancing the global public health agenda for NAFLD: a consensus statement". Nature Reviews Gastroenterology & Hepatology. 19 (1): 60–78. doi:10.1038/s41575-021-00523-4. ISSN 1759-5053. {{cite journal}}: Check date values in: |date= (help)
  24. ^ Schaffner, Monika; Rochau, Ursula; Stojkov, Igor; Qerimi Rushaj, Vjollca; Völzke, Henry; Marckmann, Georg; Lazarus, John H.; Oberaigner, Wilhelm; Siebert, Uwe (2021-04-01). "Barriers Against Prevention Programs for Iodine Deficiency Disorders in Europe: A Delphi Study". Thyroid. 31 (4): 649–657. doi:10.1089/thy.2020.0065. ISSN 1050-7256.
  25. ^ Pottie, Kevin; Hui, Charles; Rahman, Prinon; Ingleby, David; Akl, Elie A.; Russell, Grant; Ling, Li; Wickramage, Kolitha; Mosca, Davide; Brindis, Claire D. (2017-02). "Building Responsive Health Systems to Help Communities Affected by Migration: An International Delphi Consensus". International Journal of Environmental Research and Public Health. 14 (2): 144. doi:10.3390/ijerph14020144. ISSN 1660-4601. PMC 5334698. PMID 28165380. {{cite journal}}: Check date values in: |date= (help)CS1 maint: PMC format (link) CS1 maint: unflagged free DOI (link)
  26. ^ Lazarus, Jeffrey V.; Safreed-Harmon, Kelly; Kamarulzaman, Adeeba; Anderson, Jane; Leite, Ricardo Baptista; Behrens, Georg; Bekker, Linda-Gail; Bhagani, Sanjay; Brown, Darren; Brown, Graham; Buchbinder, Susan; Caceres, Carlos; Cahn, Pedro E.; Carrieri, Patrizia; Caswell, Georgina (2021-07-16). "Consensus statement on the role of health systems in advancing the long-term well-being of people living with HIV". Nature Communications. 12 (1): 4450. doi:10.1038/s41467-021-24673-w. ISSN 2041-1723. PMC 8285468. PMID 34272399.{{cite journal}}: CS1 maint: PMC format (link)
  27. ^ Moher, D; Schulz, KF; Simera, I; Altman, DG (2010). "Guidance for developers of health research reporting guidelines". PLOS Medicine. 7 (2): e1000217. doi:10.1371/journal.pmed.1000217. PMC 2821895. PMID 20169112.{{cite journal}}: CS1 maint: unflagged free DOI (link)
  28. ^ Wang, X; Chen, Y; Yang, N; Deng, W; Wang, Q; Li, N; Yao, L; Wei, D; Chen, G; Yang, K (2015). "Methodology and reporting quality of reporting guidelines: systematic review". BMC Medical Research Methodology. 15 (74): 74. doi:10.1186/s12874-015-0069-z. PMC 4579604. PMID 26395179.{{cite journal}}: CS1 maint: unflagged free DOI (link)
  29. ^ Banno, M; Tsujimoto, Y; Kataoka, Y (2020). "The majority of reporting guidelines are not developed with the Delphi method: a systematic review of reporting guidelines". Journal of Clinical Epidemiology. 124: 50–57. doi:10.1016/j.jclinepi.2020.04.010. PMID 32302679. S2CID 215809096.
  30. ^ Petzold, Axel; Biousse, Valerie; Bursztyn, Lulu; Costello, Fiona; Crum, Alison; Digre, Kathleen; Fraser, Clare; Fraser, J. Alex; Katz, Bradley; Jurkute, Neringa; Newman, Nancy; Lautrup-Battistini, Jette; Lawlor, Mitchell; Liskova, Petra; Lorenz, Birgit; Malmqvist, Lasse; Peragallo, Jason; Sibony, Patrick; Subramanian, Prem; Rejdak, Robert; Nowomiejska, Katarzyna; Touitou, Valerie; Warner, Judith; Wegener, Marianne; Wong, Sui; Yu-Wai-Man, Patrick; Hamann, Steffen (16 July 2020). "Multirater Validation of Peripapillary Hyperreflective Ovoid Mass-like Structures (PHOMS)". Neuro-Ophthalmology. 44 (6): 413–414. doi:10.1080/01658107.2020.1760891. PMC 7746256. PMID 33408429.
  31. ^ Turoff, Murray. "The Design of a Policy Delphi" (PDF). Technological Forecasting and Social Change. 2 (2): 1970.
  32. ^ Adler, Michael; Ziglio, Erio, eds. (1996), Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, London: Kingsley Publishers, ISBN 978-1-85302-104-6
  33. ^ Maurizio Bolognini (2001), Democrazia elettronica. Metodo Delphi e politiche pubbliche (Electronic Democracy. Delphi Method and Public Policy-Making) (in Italian), Rome: Carocci Editore, ISBN 978-88-430-2035-5. An example of e-democracy application is DEMOS (Delphi Mediation Online System), whose prototype was presented at the 3rd Worldwide Forum on Electronic Democracy, in 2002.
  34. ^ Tapio, P (2003). "Disaggregative Policy Delphi: Using cluster analysis as a tool for systematic scenario formation". Technological Forecasting and Social Change. 70 (1): 83–101. doi:10.1016/S0040-1625(01)00177-9.
  35. ^ Seker, Sadi Evren (2015). "Computerized Argument Delphi Technique". IEEE Access. 3 (2): 368–380. doi:10.1109/ACCESS.2015.2424703.
  36. ^ The Delphi Method:Techniques and Applications,Harold A. Linstone and Murray Turoff, Editors © 2002, Murray Turoff and Harold Linstone, TOC III.B.3. The National Drug-Abuse Policy Delphi: Progress Report and Findings to Date, IRENE ANNE JILLSON {http://is.njit.edu/pubs/delphibook/ch3b3.html
  37. ^ Green, K. C., Armstrong, J. S., & Graefe, A. (2007). Methods to elicit forecasts from groups: Delphi and prediction markets compared. Foresight: The International Journal of Applied Forecasting, 8, 17–20. [6]
  38. ^ Prokesch, Tobias; von der Gracht, Heiko; Wohlenberg, Holger (2015). "Integrating prediction market and Delphi methodology into a foresight support system — Insights from an online game". Technological Forecasting and Social Change. 97: 47–64. doi:10.1016/j.techfore.2014.02.021.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Delphi_method&oldid=1128599456"