Objective. Uncertainty analysis (UA) is an important part of simulation model validation. However, literature is imprecise as to how UA should be performed in the context of population-based microsimulation (PMS) models. In this expository paper, we discuss a practical approach to UA for such models. Methods. By adapting common concepts from published UA guidelines, we developed a comprehensive, step-by-step approach to UA in PMS models, including sample size calculation to reduce the computational time. As an illustration, we performed UA for POHEM-OA, a microsimulation model of osteoarthritis (OA) in Canada. Results. The resulting sample size of the simulated population was 500,000 and the number of Monte Carlo (MC) runs was 785 for 12-hour computational time. The estimated 95% uncertainty intervals for the prevalence of OA in Canada in 2021 were 0.09 to 0.18 for men and 0.15 to 0.23 for women. The uncertainty surrounding the sex-specific prevalence of OA increased over time. Conclusion. The proposed approach to UA considers the challenges specific to PMS models, such as selection of parameters and calculation of MC runs and population size to reduce computational burden. Our example of UA shows that the proposed approach is feasible. Estimation of uncertainty intervals should become a standard practice in the reporting of results from PMS models. 1. Introduction Computer simulation models are widely used in public health research [1, 2]. Population-based microsimulation (PMS) models are increasingly used to model possible effects of public health interventions at the population level [3–5]. Such models usually represent the population of a country: incorporate multiple cohorts, and model births, deaths, and migration [6–8]. Population-based models differ from models commonly used in cohort-based cost-effectiveness studies that model a single cohort of patients [9]. Unlike macrolevel simulation models (e.g., cell-based [10] or compartmental models [11]), microsimulation (MS) models generate a life history for every individual in a population [12, 13] and provide population-level outcomes by aggregating the individuals’ event histories [14, 15]. In PMS models of chronic, noncommunicable diseases, individuals can be treated as independent units (no interindividual interactions). Examples include models of breast cancer [7, 16], stroke [6, 17], pulmonary disease [18], colon cancer [19], diabetes [20, 21], and other chronic conditions [5, 15]. MS models that incorporate interactions between individuals, often referred to as agent-based models, have been
References
[1]
S. K. Lhachimi, W. J. Nusselder, H. C. Boshuizen, and J. P. Mackenbach, “Standard tool for quantification in health impact assessment. A review,” American Journal of Preventive Medicine, vol. 38, no. 1, pp. 78–84, 2010.
[2]
M. C. Weinstein, “Recent developments in decision-analytic modelling for economic evaluation,” PharmacoEconomics, vol. 24, no. 11, pp. 1043–1053, 2006.
[3]
E. Ackerman, “Simulation of micropopulations in epidemiology: tutorial 1—simulation: an introduction,” International Journal of Bio-Medical Computing, vol. 36, no. 3, pp. 229–238, 1994.
[4]
E. J. Feuer, R. Etzioni, K. A. Cronin, and A. Mariotto, “The use of modeling to understand the impact of screening on US mortality: examples from mammography and PSA testing,” Statistical Methods in Medical Research, vol. 13, no. 6, pp. 421–442, 2004.
[5]
B. P. Will, J. M. Berthelot, K. M. Nobrega, W. Flanagan, and W. K. Evans, “Canada's Population Health Model (POHEM): a tool for performing economic evaluations of cancer control interventions,” European Journal of Cancer, vol. 37, no. 14, pp. 1797–1804, 2001.
[6]
J. N. Struijs, M. L. L. van Genugten, S. M. A. A. Evers, A. J. H. A. Ament, C. A. Baan, and G. A. M. van den Bos, “Modeling the future burden of stroke in the Netherlands: impact of aging, smoking, and hypertension,” Stroke, vol. 36, no. 8, pp. 1648–1655, 2005.
[7]
J. Mandelblatt, C. B. Schechter, W. Lawrence, B. Yi, and J. Cullen, “The SPECTRUM population model of the impact of screening and treatment on U.S. breast cancer trends from 1975 to 2000: principles and practice of the model methods,” Journal of the National Cancer Institute Monographs, no. 36, pp. 47–55, 2006.
[8]
M. C. Wolfson, “POHEM—a framework for understanding and modelling the health of human populations,” World Health Statistics Quarterly, vol. 47, no. 3-4, pp. 157–176, 1994.
[9]
A. Brennan, S. E. Chick, and R. Davies, “A taxonomy of model structures for economic evaluation of health technologies,” Health Economics, vol. 15, no. 12, pp. 1295–1310, 2006.
[10]
B. Nosyk, B. Sharif, H. Sun, C. Cooper, and A. H. Anis, “The cost-effectiveness and value of Information of three influenza vaccination dosing strategies for individuals with human immunodeficiency virus,” PLoS ONE, vol. 6, no. 12, Article ID e27059002705, 2011.
[11]
S. Zhao, Z. Xu, and Y. Lu, “A mathematical model of hepatitis B virus transmission and its application for vaccination strategy in China,” International Journal of Epidemiology, vol. 29, no. 4, pp. 744–752, 2000.
[12]
D. Wolf, “The role of microsimulation in longitudinal data analysis. Papers in microsimulation series,” Canadian Studies in Population, vol. 28, no. 2, pp. 313–339, 2001.
[13]
C. M. Rutter, A. M. Zaslavsky, and E. J. Feuer, “Dynamic microsimulation models for health outcomes: a review,” Medical Decision Making, vol. 31, no. 1, pp. 10–18, 2011.
[14]
N. Anders Klevmarken, “Statistical inference in micro-simulation models: incorporating external information,” Mathematics and Computers in Simulation, vol. 59, no. 1–3, pp. 255–265, 2002.
[15]
R. T. Hoogenveen, P. H. M. van Baal, H. C. Boshuizen, and T. L. Feenstra, “Dynamic effects of smoking cessation on disease incidence, mortality and quality of life: the role of time since cessation,” Cost Effectiveness and Resource Allocation, vol. 6, article 1, 2008.
[16]
D. A. Berry, K. A. Cronin, S. K. Plevritis et al., “Effect of screening and adjuvant therapy on mortality from breast cancer,” The New England Journal of Medicine, vol. 353, no. 17, pp. 1784–1792, 2005.
[17]
H. J. Smolen, D. J. Cohen, G. P. Samsa et al., “Development, validation, and application of a microsimulation model to predict stroke and mortality in medically managed asymptomatic patients with significant carotid artery stenosis,” Value in Health, vol. 10, no. 6, pp. 489–497, 2007.
[18]
M. Hoogendoorn, M. P. M. H. Rutten-van M？lken, R. T. Hoogenveen et al., “A dynamic population model of disease progression in COPD,” European Respiratory Journal, vol. 26, no. 2, pp. 223–233, 2005.
[19]
F. Loeve, R. Boer, G. J. van Oortmarssen, M. van Ballegooijen, and J. D. F. Habbema, “The MISCAN-COLON simulation model for the evaluation of colorectal cancer screening,” Computers and Biomedical Research, vol. 32, no. 1, pp. 13–33, 1999.
[20]
A. J. Palmer, S. Roze, W. J. Valentine et al., “Validation of the CORE diabetes model against epidemiological and clinical studies,” Current Medical Research and Opinion, vol. 20, supplement 1, pp. S27–S40, 2004.
[21]
R. Kahn, P. Alperin, D. Eddy et al., “Age at initiation and frequency of screening to detect type 2 diabetes: a cost-effectiveness analysis,” The Lancet, vol. 375, no. 9723, pp. 1365–1374, 2010.
[22]
S. F. Railsback, S. L. Lytinen, and S. K. Jackson, “Agent-based simulation platforms: review and development recommendations,” Simulation, vol. 82, no. 9, pp. 609–623, 2006.
[23]
D. Perrin, H. J. Ruskin, and M. Crane, “Model refinement through high-performance computing: an agent-based HIV example,” Immunome Research, vol. 6, supplement 1, article S3, 2010.
[24]
R. G. Sargent, “Verification and validation of simulation models,” in Proceedings of the 37th Winter Simulation Conference (WSC '05), pp. 130–143, December 2005.
[25]
J. A. Kopec, P. Finès, D. G. Manuel et al., “Validation of population-based disease simulation models: a review of concepts and methods,” BMC Public Health, vol. 10, article 710, 2010.
[26]
C. F. Citro and E. A. Hanushek, Improving Information for Social Policy Decisions: The Uses of Microsimulation Modeling, National Research Council, National Academy Press, New York, NY, USA, 1991.
[27]
J. A. Salomon, C. D. Mathers, C. J. L. Murray, and B. Ferguson, “Methods for life expectancy and healthy life expectancy uncertainty analysis,” GPE Discussion Paper, World Health Organization, Geneva, Switzerland, August 2001, http://www.who.int/healthinfo/paper10.pdf.
[28]
A. H. Briggs, A. E. Ades, and M. J. Price, “Probabilistic sensitivity analysis for decision trees with multiple branches: use of the dirichlet distribution in a Bayesian framework,” Medical Decision Making, vol. 23, no. 4, pp. 341–350, 2003.
[29]
S. Griffin, K. Claxton, N. Hawkins, and M. Sculpher, “Probabilistic analysis and computationally expensive models: necessary and required?” Value in Health, vol. 9, no. 4, pp. 244–252, 2006.
[30]
M. G. Morgan and M. Henrion, Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press, Cambridge, UK, 1992.
[31]
K. A. Cronin, J. M. Legler, and R. D. Etzioni, “Assessing uncertainty in microsimulation modelling with application to cancer screening interventions,” Statistics in Medicine, vol. 17, no. 21, pp. 2509–2523, 1998.
[32]
S. R. Poulter, “Monte Carlo simulation in environmental risk assessment—science, policy and legal Issues,” Risk: Health, Safety & Environment, vol. 7, pp. 7–26, 1998.
[33]
G. Baio and A. P. Dawid, “Probabilistic sensitivity analysis in health economics,” Statistical Methods in Medical Research, http://www.ucl.ac.uk/statistics/research/pdfs/rr292.pdf. In press.
[34]
K. Claxton, M. Sculpher, C. McCabe et al., “Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra,” Health Economics, vol. 14, no. 4, pp. 339–347, 2005.
[35]
J. Hay and J. Jackson, “Panel 2: methodological issues in conducting pharmacoeconomic evaluations—modeling studies,” Value in Health, vol. 2, no. 2, pp. 78–81, 1999.
[36]
B. G. Koerkamp, M. C. Weinstein, T. Stijnen, M. H. Heijenbrok-Kal, and M. G. M. Hunink, “Uncertainty and patient heterogeneity in medical decision models,” Medical Decision Making, vol. 30, no. 2, pp. 194–205, 2010.
[37]
M. C. Weinstein, B. O'Brien, J. Hornberger et al., “Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR task force on good research practices—modeling studies,” Value in Health, vol. 6, no. 1, pp. 9–17, 2003.
[38]
P. Doubilet, C. B. Begg, M. C. Weinstein, P. Brun, and B. Mcneil, “Probabilistic sensitivity analysis using Monte Carlo simulation. A practical approach,” Medical Decision Making, vol. 5, no. 2, pp. 157–177, 1985.
[39]
A. O'Hagan, M. Stevenson, and J. Madan, “Monte Carlo probabilistic sensitivity analysis for patient level simulation models: efficient estimation of mean and variance using ANOVA,” Health Economics, vol. 16, no. 10, pp. 1009–1023, 2007.
[40]
D. E. Burmaster and P. D. Anderson, “Principles of good practice for the use of Monte Carlo techniques in human health and ecological risk assessments,” Risk Analysis, vol. 14, no. 4, pp. 477–481, 1994.
[41]
M. D. Stevenson, J. Oakley, and J. B. Chilcott, “Gaussian process modeling in conjunction with individual patient simulation modeling: a case study describing the calculation of cost-effectiveness ratios for the treatment of established osteoporosis,” Medical Decision Making, vol. 24, no. 1, pp. 89–100, 2004.
[42]
J. E. Oakley and A. O'Hagan, “Probabilistic sensitivity analysis of complex models: a Bayesian approach,” Journal of the Royal Statistical Society Series B, vol. 66, no. 3, pp. 751–769, 2004.
[43]
B. P. Will, K. M. Nobrega, J. M. Berthelot et al., “First do no harm: extending the debate on the provision of preventive tamoxifen,” British Journal of Cancer, vol. 85, no. 9, pp. 1280–1288, 2001.
[44]
M. D. McKay, R. J. Beckman, and W. J. Conover, “Comparison of three methods for selecting values of input variables in the analysis of output from a computer code,” Technometrics, vol. 21, no. 2, pp. 239–245, 1979.
[45]
A. Saltelli, M. Ratto, T. Andres et al., Global Sensitivity Analysis: The Primer, John Wiley & Sons, Chichester, UK, 2008.
[46]
J. A. Kopec, E. C. Sayre, W. M. Flanagan et al., “Development of a population-based microsimulation model of osteoarthritis in Canada,” Osteoarthritis and Cartilage, vol. 18, no. 3, pp. 303–311, 2010.
[47]
A. Briggs, K. Claxton, and M. Sculpher, Decision Modeling for Health Economic Evaluation. Handbooks in Health Economic Evaluation, Oxford University Press, London, UK, 2006.
[48]
C. M. Rutter, D. L. Miglioretti, and J. E. Savarino, “Bayesian calibration of microsimulation models,” Journal of the American Statistical Association, vol. 104, no. 488, pp. 1338–1350, 2009.
[49]
Statistics Canada, “National population health survey data (NPHS),” Survey information, 2011, http://www23.statcan.gc.ca:81/imdb/p2SV.pl?Function=getSurvey&SDDS=3225&lang=en&db=imdb&adm=8&dis=2.
[50]
Statistics Canada, “Canadian community health survey (CCHS) cycle 1.1,” Public use micro-data file, 2011, http://www.statcan.gc.ca/cgi-bin/imdb/p2SV.pl?Function=getSurvey&SurvId=3226&SurvVer=0&InstaId=15282&InstaVer=1&SDDS=3226&lang=en&db=imdb&adm=8&dis=2.
[51]
National Cancer Institute, “Surveillance epidemiology and end results (SEER),” US National Institute of Health, 2011, http://seer.cancer.gov/popdata/download.html.
[52]
“Population Data BC, (British Columbia, Canada)-Multi-university platform,” British Columbia’s pan-provincial population health data services, 2011, http://www.popdata.bc.ca.
[53]
A. I. Adler, “UKPDS—modelling of cardiovascular risk assessment and lifetime simulation of outcomes,” Diabetic Medicine, vol. 25, supplement 2, pp. 41–46, 2008.
[54]
S. K. Lhachimi, W. J. Nusselder, P. van Baal, et al., “DYNAMO-HIA: a dynamic microsimulation model-model specification for a dynamic model for health impact assessment,” User Guide and Manual-3rd EUPHA Conference, Amsterdam, The Netherlands, 2010, http://www.dynamo-hia.eu/object_binary/o2925_Simulation Brief.pdf.
[55]
F. Sassi, M. Cecchini, J. Lauer, and D. Chisholm, “Improving lifestyles, tackling obesity: the health and economic impact of prevention strategies,” OECD Health Working Papers 48, OECD, 2009, http://dx.doi.org/10.1787/220087432153.
[56]
H. Blossfeld and G. Rohwer, Techniques of Event History Modeling: New Approaches to Causal Analysis, Lawrence Erlbaum Associates, New Jersey, NJ, USA, 2nd edition, 2002.
[57]
D. T. Levy, P. L. Mabry, Y. C. Wang et al., “Simulation models of obesity: a review of the literature and implications for research and policy,” Obesity Reviews, vol. 12, no. 5, pp. 378–394, 2011.
[58]
M. Hiligsmann, O. Ethgen, O. Bruyère, F. Richy, H. J. Gathon, and J. Y. Reginster, “Development and validation of a markov microsimulation model for the economic evaluation of treatments in osteoporosis,” Value in Health, vol. 12, no. 5, pp. 687–696, 2009.
[59]
Statistics Canada, “The lifepaths microsimulation model (version 1.1): an overview. Information of the model,” 2011, http://www.statcan.gc.ca/spsd/LifePathsOverview_E.pdf.
[60]
G. Rowe and H. Nguyen, “Longitudinal analysis of labour force survey data,” Survey Methodology (Statistics Canada), vol. 30, no. 1, pp. 105–114, 2004.
[61]
F. Willekens, “Continuous-time microsimulation in longitudinal analysis,” in New Frontiers in Microsimulation Modelling, A. Zaidi, A. Harding, and P. Williamson, Eds., pp. 413–436, Ashgate, Surrey, UK, 2009.
[62]
R. Bender, T. Augustin, and M. Blettner, “Generating survival times to simulate Cox proportional hazards models,” Statistics in Medicine, vol. 24, no. 11, pp. 1713–1723, 2005.
[63]
M. E. Kuhl, J. S. Ivy, E. K. Lada, N. M. Steiger, M. A. Wagner, and J. R. Wilson, “Univariate input models for stochastic simulation,” Journal of Simulation, vol. 4, no. 2, pp. 81–97, 2010.
[64]
P. H. Garthwaite, J. B. Kadane, and A. O'Hagan, “Statistical methods for eliciting probability distributions,” Journal of the American Statistical Association, vol. 100, no. 470, pp. 680–701, 2005.
[65]
D. J. Pasta, J. L. Taylor, and J. M. Henning, “Probabilistic sensitivity analysis incorporating the bootstrap: an example comparing treatments for the eradication of Helicobacter pylori,” Medical Decision Making, vol. 19, no. 3, pp. 353–363, 1999.
[66]
D. Yeo, H. Mantel, and T. P. Liu, “Bootstrap variance estimation for the National Population Health Survey,” Proceedings of Survey Research Methods Section, The American Statistical Association, vol. 12, no. 1, pp. 778–785, 1999.
[67]
R. Dowling, A. Skabardonis, J. Halkias, G. McHale, and G. Zammit, “Guidelines for calibration of microsimulation models: framework and applications,” Transportation Research Record, vol. 1876, no. 1, pp. 1–9, 2004.
[68]
G. Parmigiani, “Measuring uncertainty in complex decision analysis models,” Statistical Methods in Medical Research, vol. 11, no. 6, pp. 513–537, 2002.
[69]
D. Draper, “Assessment and propagation of model uncertainty,” Journal of the Royal Statistical Society Series B, vol. 57, pp. 45–97, 1995.
[70]
A. M. Zaslavsky and S. W. Thurston, “Error analysis of food stamp microsimulation models: further results,” in Proceedings of the ASA Social Statistics Section, pp. 151–156, American Statistical Association, Alexandria, Va, USA, 1995.
