Toolcatalogus

References

Handbooks:

B.M. Ayyub, (2001) Elicitation of Expert Opinions for Uncertainty and Risks, CRC Press, Florida.

T. Bedford, R. Cooke (2001). Probabilistic Risk Analysis: Foundations and Methods.

Cambridge University Press.

R.M. Cooke. Experts in uncertainty: Opinion and Subjective Probability in Science. New York, Oxford University Press, 1991.

Cullen, A.C. and H.C. Frey, H.C. (1999). Probabilistic Techniques in Exposure Assessment, Plenum Publishing Corp., New York, USA.

R. Dawes, (1990) Rational Choice in an Uncertain World.

D. Griffin, T. Gilovich, D. Kahneman (eds.) (2002) Heuristics and Biases: Psychology of Intuitive Judgment.

D. Kahneman, A. Tversky, P. Slovic (eds.) [1982]. Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press.

M.A. Meyer, J.M. Booker (1991). Eliciting and Analyzing Expert Judgement: A practical Guide, Academic Press, London

Papers and Reports:

J.L. Anderson (1998) Embracing Uncertainty: The Interface of Bayesian Statistics and Cognitive Psychology. Conservation Ecology Vol. 2 (cf. http://www.consecol.org/Journal/vol2/iss1/art2/)

J.L. Anderson [1998] Enhancing Communication About Uncertainty. Extension Note, British Columbia; July 1998;

G.B. Baecher, (2002). Expert elicitation in geotechnical risk assessments, (http:/www.glue.umd.edu/~gbaecher/papers.d/judge_prob.d/judge_prob.html)

G.B. Baecher, and J.T. Christian (2000). "Natural variation, limited knowledge, and the nature of uncertainty in risk analysis". Presented at 'Risk-based Decisionmaking in Water Resources IX', Oct. 15-20, 2000, Santa Barbara. http://www.glue.umd.edu/~gbaecher/papers.d/Baecher_&_Christian Sta Barbara 2000.pdf

M. Benarie (1988) Delphi- and delphilike approaches with special regard to environmental standard setting. Technological Forecasting and Social Change, Vol. 33, pp. 149-158.

R.T. Clemen, K.C. Lichtendahl, jr. 2002 Debiasing Expert Confidence: A Bayesian Calibration Model, presented at PSAM6-conference

R.T. Clemen, T. Reilly [1999] Correlations and copulas for decision and risk analysis.

Management Science. 45: pp. 208-224.

R.T. Clemen, R.L. Winkler, 1999 Combining probability distributions from experts in risk analysis. Risk Analysis, Vol. 19, pp. 187-203

R.M. Cooke (1995). UNICORN; Methods and Code for Uncertainty Analysis. Published by the Atomic Energy Association, Delft University of Technology.

R.M. Cooke, L.J.H. Goosens, (2000a). Procedures Guide for Structured Expert Judgment. Technical report EUR 18820 EN, European Commission, Directorate-General for Research, Brussels, Belgium, 2000

R.M. Cooke, L.H.J. Goossens (2000b) Procedures Guide for Structure Expert Judgement in

Accident Consequence Modelling. Radiation Protection and Dosimetry vol 90, no. 3 2000 pp 303-311

R. Cooke, B. Kraan (2000) Processing expert judgements in accident consequence modelling. Radiation Protection Dosimetry. Vol. 90, No. 3, pp. 311-315.

R. Cooke, A. Meeuwissen, 1994 Tree dependent random variables. Technical Report 94-28. Dept. of Mathematics, Delft University of Technology.

Cosmides L. en J. Tooby [1996] Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgement under uncertainty. Cognition, Vol. 58, pp. 1-73.

P. Embrechts, A. McNeil, D. Straumann (1999). Correlation and Dependence in Risk

Management: Properties and pitfalls, To appear in Risk Management: Value at Risk and Beyond, ed. By M. Dempster, H.K. Moffatt. Cambrdige University Press, 2001

S. Ferson, L.R. Ginzburg, 1996. Different methods are needed to propagate ignorance and

variability. Reliability Engineering and System Safety Vol. 54, pp. 133 - 144.

H.C. Frey, 1998, BRIEFING PAPER PART 1: Introduction to Uncertainty Analysis. (http://courses.ncsu.edu/classes/ce456001/www/Background1.html)

Gigerenzer, 1991 How to make cognitive illusions disappear: Beyond 'heuristics and biases'. European Review of Social Psychology, Vol. 2, 83-115.

Gigerenzer 1994, Why the distinction between single event probabilities and frequencies is relevant for psychology (and vice versa).

L. Goossens, R. Cooke, B. Kraan (1998) Evaluation of weighting schemes for expert judgment studies. PSAM 4 Proceedings, (Moselh and Bari eds) Springer 1998,1937-1942.

A O'Hagan (1998) Eliciting expert beliefs in substantial practical applications. The Statistician, Vol. 47, pp. 21-35

E. Hofer (1996) When to Separate Uncertainties and When Not to Separate. Reliability Engineering and System Safety Vol. 54, pp. 113 - 118

F.O. Hoffman, J.S. Hammonds (1994) Propagation of uncertainty in risk assessment: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. Risk Analysis, Vol. 14, pp. 707-712.

Hoffrage U., Lindsey, S., Hertwig R., Gigerenzer G. [2000] Communicating Statistical Information. Science Vol. 290, pp. 2261 e.v.

S.C. Hora (1992) Acquisition of Expert Judgment: Examples from Risk Assessment. Journal of Energy Engineering, Vol. 118, pp. 136-148.

S.C. Hora (1996) Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety, Vol. 54, pp. 217-223

S.C. Hora and R.L. Iman (1989) Expert opinion in risk analysis: The NUREG-1150 Methodology. Nuclear Science and Engineering, Vol. 102, pp. 323-331.

J.B. Kadane, L.J.Wolfson, P.S. Craig, M. Goldstein, A.H. Seheult, J.A. Smith, A. O'Hagan (1998) Papers on 'Elicitation'. The Statistician. Vol. 47, Part 1, pp. 3-68

R.L. Keeney, D. van Winterfeldt (1991) Eliciting probabilities from experts in complex technical problems. IEEE Trans. On Engineering Management. Vol. 38, pp. 191-201.

D.W. Keith (1996). When is it appropriate to combine expert judgements? Climatic Change, Vol. 33, pp. 139-143.

B. Kraan, (2002). Probabilistic Inversion in Uncertainty Analysis. PH-D. Technical University Delft.

D. Kurowicka, R. Cooke, 2001 Conditional, partial and rank correlation for elliptical copula. In Dependence Modeling in Uncertainty Analysis, Proc. Of ESREL 2001, pp. 259-276.

Lau, A. H. and Leong, T. Y. (1999) PROBES: A Framework for probabilities elicitation from experts. In Proceedings of the 1999 AMIA Annual Fall Symposium, pages 301-305, AMIA. See http://www.amia.org/pubs/symposia/D005714.PDF

J. Li, A. Dekhtyar, J. Goldsmith, 2002: Efficiently eliciting many probabilities online. See http://www.cs.uky.edu/~dekhtyar/dblab/hypo.ldg.pdf

M.W. Merkhofer (1987) Quantifying judgmental uncertainty: methodology, experiences and insights. IEEE Trans. On Systems, Man and Cybernetics. Vol. SMC-17, pp.741-752.

M.G. Morgan M. Henrion (1990) Uncertainty, A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press.

M.G. Morgan and D. Keith (1995), Subjective Judgments by Climate experts. Environmental Science & Technology, Vol. 29, pp. 468-476.

M.G. Morgan, L.F. Pitelka, E. Shevliakova (2001) Eastimates of Climate Change impacts on forest ecosystems. Climatic Change, Vol. 49, pp. 279-307.

M.G. Morgan (2003), Characterizing and Dealing with Uncertainty: Insights from the Integrated Assessment of Climate Change. Integrated Assessment. Vol. 4, pp. 46-55.

Nauta, M, (2001) Risk assessment of Shiga-toxin producing Escherichia coli O157 in steak

tartare in the Netherlands. RIVM Report 257851003

NCRP, 1996 A Guide for Uncertainty Analysis in Dose and Risk Assessments related to

Environmental Contamination NCRP Commentary no. 14.

W.D. Nordhaus (1994) Expert opinion on climate change. American Scientist, Vol. 82, pp. 45-51.

N.R. Ortiz, T.A. Wheeler, R.J. Breeding, S. Hora, M.A. Meyer, R.L. Keeney (1991) Use of expert judgment in NUREG-1150. Nuclear Engineering and Design, Vol. 126, pp. 313-331.

H. Otway, D. van Winterfeldt (1992) Expert judgment in risk analysis and management: process, context and pitfalls. Risk Analysis, Vol. 12, pp. 83-93.

P. Reichert, D.W. Keith [2003] Bayesian Combination of Expert Judgements: Reinterpretation

 of the Meaning and dependence of elicited distributions. Submitted for publication.

J.S. Risbey, J.P. van der Sluijs, and J. Ravetz, 2001, A Protocol for Assessment of Uncertainty and Strength of Emissions Data. Copernicus Institute, University Utrecht; NW&S E-2001-10

 F.A. Seiler, J.L. Alvarez, 1996. On the selection of distributions for stochastic variables.

Risk Analysis, Vol. 16. pp. 5-18.

J.P. van der Sluijs, J.S. Risbey and J. Ravetz, 2003, Uncertainty Assessment of VOC emissions from Paint in the Netherlands. Copernicus Institute, University Utrecht; NW&S E-2002-13

C.S. Spetzler, and S. von Holstein (1975). "Probability Encoding in Decision Analysis." Management Science, 22(3).

A. Tversky, D. Kaheneman (1974) Judgment under uncertainty: heuristics and biases. Science, Vol. 185, pp. 1124-1131.

I.H.J. Van der Fels-Klerx, Cooke, R.M., Nauta, M.J., Goossens, L.H.J., Havelaar, A.H., 2004.

A structured expert judgment study for a model of campylobacter contamination during broiler chicken processing. Submitted to Risk Analysis

I.H.J. Van Der Fels-Klerx, L.H.J. Goossens, H.W. Saatkamp, S.H.S. Horst (2002) Elicitation of quantitative data from a heterogeneous expert panel: formal process and application in animal health. Risk Analysis. Vol. 22, pp. 67-81.

J. van Lenthe (1993) A blueprint of ELI: a new method for eliciting subjective probability distributions. Behavior Research Methods, Instrum. & Computers. Vol. 25, pp. 425-433.

M.M.D. van Oorschot, B.C.P. Kraan, R.M.M. van den Brink, P.H.M. Janssen, R.M. Cooke, 2003. Uncertainty Analysis for NO_x emissions from Dutch passenger cars in 1998. Applying structured expert elicitation and distinguishing different types of uncertainty.
RIVM report 550002004.

Example case studies:

Van der Sluijs, et al. 2003: Uncertainty assessment of VOC-emissions from paint in the Netherlands.

Van Oorschot, et al. 2003: Uncertainty analysis for NO_x emissions from Dutch passenger cars in 1998.

Nauta (2001): Risk assessment for foodproduction and foodprocessing, case of steak tartare.

Van der Fels (2004): Risk assessment for campyylobacter contamination

Software:

Concerning software which can be an aid in probability elicitation:

• PROBES: A Framework for Probability Elicitation from Experts; see Lau and Leong (1999).
• HYPO: Software for elicitation of many probabilities in Bayesian network context; see Li, Dekhtyar, Goldsmith (2002)
• ELI: Software foreliciting subjective probability distributions; see van Lenthe, 1993.
• EXCALIBUR: Software for expert calibration.
• UNICORN: software for uncertainty analysis with correlations

For the latter two packages, see the website: http://ssor.twi.tudelft.nl/~risk/br/software.html

Websites:

http://ssor.twi.tudelft.nl/~risk/br/software.html: Software for expert calibration (EXCALIBUR) and for uncertainty analysis with correlations (UNICORN) http://courses.ncsu.edu/classes/ce456001/www/Background1.html: primer on expert elicitation

Experts at PBL:

Mark van Oorschot, Peter Janssen