38 www.e-enm.org

Endocrinol Metab 2016;31:38-44 http://dx.doi.org/10.3803/EnM.2016.31.1.38 pISSN 2093-596X · eISSN 2093-5978

Review Article

How to Establish Clinical Prediction Models Yong-ho Lee1, Heejung Bang2, Dae Jung Kim3

1Department of Internal Medicine, Yonsei University College of Medicine, Seoul, Korea; 2Division of Biostatistics, Department of Public Health Sciences, University of California Davis School of Medicine, Davis, CA, USA; 3Department of Endocrinology and Metabolism, Ajou University School of Medicine, Suwon, Korea

A clinical prediction model can be applied to several challenging clinical scenarios: screening high-risk individuals for asymp- tomatic disease, predicting future events such as disease or death, and assisting medical decision-making and health education. Despite the impact of clinical prediction models on practice, prediction modeling is a complex process requiring careful statisti- cal analyses and sound clinical judgement. Although there is no definite consensus on the best methodology for model develop- ment and validation, a few recommendations and checklists have been proposed. In this review, we summarize five steps for de- veloping and validating a clinical prediction model: preparation for establishing clinical prediction models; dataset selection; handling variables; model generation; and model evaluation and validation. We also review several studies that detail methods for developing clinical prediction models with comparable examples from real practice. After model development and vigorous validation in relevant settings, possibly with evaluation of utility/usability and fine-tuning, good models can be ready for the use in practice. We anticipate that this framework will revitalize the use of predictive or prognostic research in endocrinology, leading to active applications in real clinical practice.

Keywords: Clinical prediction model; Development; Validation; Clinical usefulness

INTRODUCTION

Hippocrates emphasized prognosis as a principal component of medicine [1]. Nevertheless, current medical investigation mostly focuses on etiological and therapeutic research, rather than prognostic methods such as the development of clinical prediction models. Numerous studies have investigated wheth- er a single variable (e.g., biomarkers or novel clinicobiochemi- cal parameters) can predict or is associated with certain out-

comes, whereas establishing clinical prediction models by in- corporating multiple variables is rather complicated, as it re- quires a multi-step and multivariable/multifactorial approach to design and analysis [1]. Clinical prediction models can inform patients and their physicians or other healthcare providers of the patient’s proba- bility of having or developing a certain disease and help them with associated decision-making (e.g., facilitating patient-doc- tor communication based on more objective information). Ap-

Received: 9 January 2016, Revised: 14 January 2016, Accepted: 27 January 2016 Corresponding authors: Dae Jung Kim Department of Endocrinology and Metabolism, Ajou University School of Medicine, 164 World cup-ro, Yeongtong-gu, Suwon 16499, Korea Tel: +82-31-219-5128, Fax: +82-31-219-4497, E-mail: [email protected]

Yong-ho Lee Department of Internal Medicine, Yonsei University College of Medicine, 50-1 Yonsei-ro, Seodaemun-gu, Seoul 03722, Korea Tel: +82-2-2228-1943, Fax: +82-2-393-6884, E-mail: [email protected]

Copyright © 2016 Korean Endocrine Society This is an Open Access article distributed under the terms of the Creative Com- mons Attribution Non-Commercial License (http://creativecommons.org/ licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribu- tion, and reproduction in any medium, provided the original work is properly cited.

Clinical Prediction Models

Copyright © 2016 Korean Endocrine Society www.e-enm.org 39

Endocrinol Metab 2016;31:38-44 http://dx.doi.org/10.3803/EnM.2016.31.1.38 pISSN 2093-596X · eISSN 2093-5978

plying a model to a real world problem can help with detection or screening in undiagnosed high-risk subjects, which improves the ability to prevent developing diseases with early interven- tions. Furthermore, in some instances, certain models can pre- dict the possibility of having future disease or provide a prog- nosis for disease (e.g., complication or mortality). This review will concisely describe how to establish clinical prediction models, including the principles and processes for conducting multivariable prognostic studies and developing and validating clinical prediction models.

CONCEPT OF CLINICAL PREDICTION MODELS

In the era of personalized medicine, prediction of prevalent or incident diseases (diagnosis) or outcomes for future disease course (prognosis) became more important for patient manage- ment by health-care personnel. Clinical prediction models are used to investigate the relationship between future or unknown outcomes (endpoints) and baseline health states (starting point) among people with specific conditions [2]. They generally combine multiple parameters to provide insight into the relative impacts of individual predictors in the model. Evidence-based medicine requires the strongest scientific evidence, including findings from randomized controlled trials, meta-analyses, and systematic reviews [3]. Although clinical prediction models are partly based on evidence-based medicine, the user must also adopt practicality and an artistic approach to establish clinically relevant and meaningful models for targeted users. Models should predict specific events accurately and be rela- tively simple and easy to use. If a prediction model provides inaccurate estimates of future-event occurrences, it will mislead healthcare professionals to provide insufficient management of patients or resources. On the other hand, if a model has high predictability power but is difficult to apply (e.g., with compli- cated calculation or unfamiliar question/item or unit), time con- suming, costly [4] or less relevant (e.g., European model for Koreans, event too far away), it will not be commonly used. For example, a diabetes prediction model developed by Lim et al. [5] has a relatively high area under the receiver operating curve (AUC, 0.77), while blood tests that measure hemoglobin A1c, high density lipoprotein cholesterol, and triglyceride are included in the risk score, which would generally require clini- cian’s involvement so could be a major barrier for use in com- munity settings. When prediction models consist of complicat- ed mathematical equations [6,7], a web-based application can

enhance implementation (e.g., calculating 10-year and lifetime risk for atherosclerotic cardiovascular disease [CVD] is avail- able at http://tools.acc.org/ASCVD-Risk-Estimator/). There- fore, achieving a balance between predictability and simplicity is a key to a good clinical prediction model.

STEPS TO DEVELOPING CLINICAL PREDICTION MODELS

There are several reports [1,8-13] and a textbook [14] that de- tail methods to develop clinical prediction models. Although there is currently no consensus on the ideal construction meth- od for prediction models, the Prognosis Research Strategy (PROGRESS) group has proposed a number of methods to im- prove the quality and impact of model development [2,15]. Re- cently, investigators on the Transparent Reporting of a multi- variable prediction model for Individual Prognosis Or Diagno- sis (TRIPOD) study have established a checklist of recommen- dations for reporting on prediction or prognostic models [16]. This review will summarize the analytic process for developing clinical prediction models into five stages.

Stage 1: preparation for establishing clinical prediction models The aim of prediction modeling is to develop an accurate and useful clinical prediction model with multiple variables using comprehensive datasets. First, we have to articulate several im- portant research questions that affect database selection and the approach of model generation. (1) What is the target outcome (event or disease) to predict (e.g., diabetes, CVD, or fracture)? (2) Who is the target patient of the model (e.g., general popula- tion, elderly population ≥65 years or patients with type 2 dia- betes)? (3) Who is the target user of the prediction model (e.g., layperson, doctor or health-related organization)? Depending on the answers to the above questions, researchers can choose the proper datasets for the model. The category of target users will determine the selection and handling process of multiple variables, which will affect the structure of the clinical predic- tion model. For example, if researchers want to make a predic- tion model for laypersons, a simple model with not many user- friendly questions in only a few categories (e.g., yes vs. no) could be ideal.

Stage 2: dataset selection The dataset is one of the most important components of the clinical prediction model—often not under investigators’ con-

Lee YH, et al.

40 www.e-enm.org Copyright © 2016 Korean Endocrine Society

trol—and ultimately determines its quality and credibility; however, there are no general rules for assessing the quality of data [9]. Yet, there is no such thing as perfect data and prefect model. It would be reasonable to search for best-suited dataset. Oftentimes, secondary or administrative data sources must be utilized because a primary dataset with the study endpoint and all of key predictors is not available. Researchers should use different types of datasets, depending on the purpose of the prediction model. For example, a model for screening high-risk individuals with undiagnosed condition/disease can be devel- oped using cross-sectional cohort data. However, such models may have relatively low power for predicting future incidence of disease when different risk factors come into play. Accord- ingly, longitudinal or prospective cohort datasets should be used for prediction models for future events (Table 1). Models for prevalent events are useful for predicting asymptomatic diseases, such as diabetes or chronic kidney disease, by screen- ing undiagnosed cases, whereas models for incident events are useful for predicting the incidence of relatively severe diseases, such as CVD, stroke, and cancer. A universal clinical prediction model for disease does not exist; thus, separate specific models that can individually as- sess the role of ethnicity, nationality, sex, or age on disease risk are warranted. For example, the Framingham coronary heart disease (CHD) risk score is generated by one of the most com- monly used clinical prediction models; however, it tends to overestimate CHD risk by approximately 5-fold in Asian popu- lations [17,18]. This indicates that models derived from one ethnicity sample may not be directly applied to populations of other ethnicities. Other specific characteristics of study popula- tions beside ethnicity (e.g., obesity- or culture-related vari- ables) could be important. There is no absolute consensus on the minimal requirement for dataset sample size. Generally, large representative, contem-

porary datasets that closely reflect the characteristics of their target population are ideal for modeling and can enhance the relevance, reproducibility, and generalizability of the model. Moreover, two types of datasets are generally needed: a devel- opment dataset and a validation dataset. A clinical prediction model is first derived from analyses of the development dataset and its predictive performance should be assessed in different populations based on the validation dataset. It is highly recom- mended to use validation datasets from external study popula- tions or cohorts, whenever available [19,20]; however, if it is not possible to find appropriate external datasets, an internal validation dataset can be formed by randomly splitting the orig- inal cohort into two datasets (if sample size is large) or statisti- cal techniques such as jackknife or bootstrap resampling (if not) [21]. The splitting ratio can vary depending on the researchers’ particular goals, but generally, more subjects should be allocat- ed to the development dataset than to the validation dataset.

Stage 3: handling variables Since cohort datasets contain more variables than can reason- ably be used in a prediction model, evaluation and selection of the most predictive and sensible predictors should be done. Generally, inclusion of more than 10 variables/questions may decrease the efficiency, feasibility and convenience of predic- tion models, but expert’s judgment that could be somewhat subjective is required to assess the need for each situation. Pre- dictors that were previously found to be significant should nor- mally be considered as candidate variables (e.g., family history of diabetes in diabetes risk score). It should be noted that not all significant predictors need to be included in the final model (e.g., P<0.05); predictor selection must be always guided by clinical relevance/judgement to prevent nonsensical or less rel- evant or user-unfriendly variables (e.g., socioeconomic status- related) or possible false-positive associations. Additionally,

Table 1. Characteristics of Different Clinical Prediction Models according to Their Purpose

Characteristic Prevalent/concurrent events Incident/future events

Data type Cross-sectional data Longitudinal/prospective cohort data

Application Useful for asymptomatic diseases for screening undiagnosed cases (e.g., diabetes, CKD)

Useful for predicting the incidence of diseases (e.g., CVD, stroke, cancer)

Aim of the model Detection Prevention

Simplicity in model and use More important Less important

Example Korean Diabetes Score [34] ACC/AHA ASCVD risk equation [7]

CKD, chronic kidney disease; CVD, cardiovascular disease; ACC/AHA, American College of Cardiology/American Heart Association; ASCVD, atherosclerotic cardiovascular disease.

Clinical Prediction Models

Copyright © 2016 Korean Endocrine Society www.e-enm.org 41

variables which are highly correlated with others may be ex- cluded because they contribute little unique information [22]. On the other hand, variables not statistically significant or with small effect size may still contribute to the model [23]. De- pending on researcher discretion, different models that analyze different variables may be developed for targeting distinct us- ers. For example, a simple clinical prediction model that does not require laboratory variables and a comprehensive model that does could both be designed for laypersons and health care providers, respectively [19]. With regard to variable coding, categorical and continuous variables should be managed differently [8]. For ordered cate- gorical variables, infrequent categories can be merged and sim- ilar variables may be combined/grouped. For example, past and current smoker categories can be merged if numbers of sub- jects who report being a past or current smoker are relatively small and variable unification does not alter the statistical sig- nificance of the model materially. Although continuous param- eters are usually included in a regression model, assuming lin- earity, researchers should consider the possibility of non-linear associations such as J- or U-shaped distributions [24]. Further- more, the relative effect of a continuous variable is determined by the measurement scale used in the model [8]. For example, the impact of fasting glucose levels on the risk of CVD may be interpreted as having a stronger influence when scaled per 10 mg/dL than per 1 mg/dL. Researchers often emphasize the importance of not dichoto- mizing continuous variables in the initial stage of model devel- opment because valuable predictive information can be lost during categorization [24]. However, prediction models—is not the same thing as regression models—with continuous pa- rameters may be complex and hard to use or be understood by laypersons, because they have to calculate their risk scores by themselves. A web or computer-based platform is usually re- quired for the implementation of these models. Otherwise, in a later phase, researchers may transform the model into a user- friendly format by categorizing some predictors, if the predic- tive capacity of the model is retained [8,19,25]. Finally, missing data is a chronic problem in most data anal- yses. Missing data can occur various reasons, including uncol- lected (e.g., by design), not available or not applicable, refusal by respondent, dropout, or “don’t know.” To handle this issue, researchers may consider imputation technique, dichotomizing the answer into yes versus others, or allow “unknown” as a separate category as in http://www.cancer.gov/bcrisktool/.

Stage 4: model generation Although there are no consensus guidelines for choosing vari- ables and determining structures to develop the final prediction model, various strategies with statistical tools are available [8,9]. Regression analyses, including linear, logistic, and Cox models are widely used depending on the model and its intend- ed purpose. First, the full model approach is to include all the candidate variables in the model; the benefit of this approach is to avoid overfitting and selection bias [9]. However, it can be impractical to pre-specify all predictors and previously signifi- cant predictors may not be in a new population/sample. Sec- ond, a backward elimination approach or stepwise selection method can be applied to remove a number of insignificant candidate variables. To check for overfitting of the model, Akaike information criterion (AIC) [26], an index of model fit- ting that charges a penalty against larger models, may be useful [19]. Lower AIC values indicate a better model fit. Some inter- pret that AIC addresses explanation and Bayesian information criterion (BIC) addresses prediction, where BIC may be con- sidered a Bayesian counterpart [27]. If researchers prefer algorithm modeling culture instead of data modeling culture, e.g., formula-based regression [28], a classification and regression tree analysis or recursive parti- tioning could be considered [28-30]. With regard to determining scores for each predictor in the generation of simplified models, researchers using expert judg- ment may create a weighted scoring system by converting β coefficients [19] or odds ratios [20] from the final model to in- teger values, while preserving monotonicity and simplicity. For example, from the logistic regression model built by Lee et al. [19], β coefficients <0.6, 0.7 to 1.3, 1.4 to 2.0, and >2.1 were assigned scores of 1, 2, 3, and 4, respectively.

Stage 5: model evaluation and validation (internal/ external) After model generation, researchers should evaluate the predic- tive power of their proposed model using an independent datas- et, where truly external dataset is preferred whenever available. There are several standard performance measures that capture different aspects: two key components are calibration and dis- crimination [8,9,31]. Calibration can be assessed by plotting the observed proportions of events against the predicted probabili- ties for groups defined by ranges of individual predicted risk [9,10]. For example, a common method is to categorize 10 risk groups of equal size (deciles) and then conduct the calibration process [32]. The most ideal calibration plot would show a 45°

Lee YH, et al.

42 www.e-enm.org Copyright © 2016 Korean Endocrine Society

line, which indicates that the observed proportions of events and predicted probabilities completely overlap over the entire range of probabilities [9]. However, this is not guaranteed when external validation is conducted with a different sample. Dis- crimination is defined as the ability to distinguish events versus non-events (e.g., dead vs. alive) [8]. The most common dis- crimination measure is the AUC or, equivalently, concordance (c)-statistic. The AUC is equal to the probability that, given two individuals randomly selected—one who will develop an event and another who will not—the model will assign a higher prob- ability of an event to the former [10]. A c-statistic value of 0.5 indicates a random chance (i.e., flip of a coin). The usual c-sta- tistic range for a prediction model is 0.6 to 0.85; this range can be affected by target-event characteristics (disease) or the study population. A model with a c-statistic ranging from 0.70 to 0.80 has an adequate power of discrimination; a range of 0.80 to 0.90 is considered excellent. Table 2 shows several common statisti- cal measures for model evaluation. As usual, selection, application and interpretation of any sta- tistical method and results need great care as virtually all meth- ods entail assumptions and limited capacity. Let us review some here. Predictive values depend on the disease prevalence so direct comparison for different diseases may not be valid. When sample size is very large, P value can be impressively small even for a practically meaningless difference. Net reclas- sification index and integrated discrimination improvement are known to lead to non-proper scoring and vulnerable to miscali-

brated or overfit problems [33]. AUC and R2 are often hard to increase by a new predictor, even with large odds ratio. Despite similar names, AIC and BIC address slightly different issues and information in BIC can be decreased with sample size increases. The Hosmer-Lemeshow test is highly sensitive when sample size is large, which is not an ideal property as a goodness-fit sta- tistic. Calibration plot can easily provide a high correlation coef- ficient (>0.9), simply because they are computed for predicted versus observed values on grouped data (without random vari- ability). Finally, AUC also needs caution: a high value (e.g., >0.9) may mean excellent discrimination but it can also reflect the situation where prediction is not so relevant: (1) the task is closer to diagnostic or early onset rather than prediction; (2) cas- es vs. non-cases are fundamentally different with minimal over- lap; or (3) predictors and endpoints are virtually the same things (e.g., current blood pressure vs. future blood pressure). Despite the long list provided above, we do not think this is a discouraging news to researchers. We may tell us no method is perfect and “one size does not fit all” is also true to statistical methods; thus blinded or automated application can be danger- ous. It is crucial to separate internal and external validation and to conduct the previously mentioned analyses on both datasets to finalize the research findings (see the following for example reports [19,20,34]). Internal validation can be done using a ran- dom subsample or different years from the development dataset or by conducting bootstrap resampling [22]. This approach can particularly assess the stability of selected predictors, as well as prediction quality. Subsequently, external validation should be performed on an independent dataset from that which was pre- viously used to develop the model. For example, datasets can be obtained from populations from other hospitals or centers (see geographic validation [19]) or a more recently collected cohort population (temporal validation [34]). This process is often considered to be a more powerful test for prediction mod- els than internal validation because it evaluates transportability, generalizability and true replication, rather than reproducibility [8]. Poor model performance may occur after use of an external dataset due to differences in healthcare systems, measurement methods/definitions of predictors and/or endpoint, subject characteristics or context (e.g., high vs. low risk).

CONCLUSIONS

For patient-centered perspectives, clinical prediction models are useful for several purposes: to screen high-risk individuals

Table 2. Statistical Measures for Model Evaluation

Sensitivity and specificity

Discrimination (ROC/AUC)

Predictive values: positive, negative

Likelihood ratio: positive, negative

Accuracy: Youden index, Brier score

Number needed to treat or screen

Calibration: Calibration plot, Hosmer-Lemeshow test

Model determination: R2

Statistical significance: P value (e.g., likelihood ratio test)

Magnitude of association, e.g., β coefficient, odds ratio

Model quality: AIC/BIC

Net reclassification index and integrated discrimination improvement

Net benefit

Cost-effectiveness

ROC, receiver operating characteristic; AUC, area under the curve; AIC, Akaike information criterion; BIC, Bayesian information criterion.

Clinical Prediction Models

Copyright © 2016 Korean Endocrine Society www.e-enm.org 43

for asymptomatic disease, to predict future events of disease or death, and to assist medical decision-making. Herein, we sum- marized five steps for developing a clinical prediction model. Prediction models are continuously designed but few have had their predictive performance validated with an external popula- tion. Because model development is complex, consultation with statistical experts can improve the validity and quality of rigorous prediction model research. After developing the mod- el, vigorous validation with multiple external datasets and ef- fective dissemination to interested parties should occur before using the model in practice [35]. Web or smartphone-based ap- plications can be good routes for advertisement and delivery of clinical prediction models to the public. For example, Korean risk models for diabetes, fatty liver, CVD, and osteoporosis are readily available at http://cmerc.yuhs.ac/mobileweb/. Simple model may be translated into a one page checklist for patient’s self-assessment (e.g., equipped in waiting room in clinic). We anticipate that the framework that we provide/summarize, along with additional assistance from related references or text- books, will help predictive or prognostic research in endocri- nology; this will lead to active application of these practices in real world settings. In light of the personalized- and precision- medicine era, further research is needed to attain individual- level predictions, where genetic or novel biomarkers can play bigger roles, as well as simple generalized predictions which can further help patient-centered care.

CONFLICTS OF INTEREST

No potential conflict of interest relevant to this article was re- ported.

ACKNOWLEDGMENTS

This study was supported by a grant from the Korea Healthcare Technology R&D Project, Ministry of Health and Welfare, Re- public of Korea (No. HI14C2476). H.B. was partly supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 TR 000002. D.K. was partly supported by a grant of the Korean Health Technology R&D Project, Ministry of Health and Welfare, Re- public of Korea (HI13C0715).

ORCID

Yong-ho Lee http://orcid.org/0000-0002-6219-4942

Dae Jung Kim http://orcid.org/0000-0003-1025-2044

REFERENCES

1. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ 2009;338:b375.

2. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, et al. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ 2013;346:e5595.

3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Rich- ardson WS. Evidence based medicine: what it is and what it isn’t. BMJ 1996;312:71-2.

4. Greenland S. The need for reorientation toward cost-effective prediction: comments on ‘Evaluating the added predictive ability of a new marker. From area under the ROC curve to re- classification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med 2008;27:199- 206.

5. Lim NK, Park SH, Choi SJ, Lee KS, Park HY. A risk score for predicting the incidence of type 2 diabetes in a middle- aged Korean cohort: the Korean genome and epidemiology study. Circ J 2012;76:1904-10.

6. Griffin SJ, Little PS, Hales CN, Kinmonth AL, Wareham NJ. Diabetes risk score: towards earlier detection of type 2 diabe- tes in general practice. Diabetes Metab Res Rev 2000;16:164- 71.

7. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Associa- tion Task Force on Practice Guidelines. Circulation 2014;129 (25 Suppl 2):S49-73.

8. Steyerberg EW, Vergouwe Y. Towards better clinical pre- diction models: seven steps for development and an ABCD for validation. Eur Heart J 2014;35:1925-31.

9. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ 2009;338:b604.

10. Altman DG, Vergouwe Y, Royston P, Moons KG. Progno- sis and prognostic research: validating a prognostic model. BMJ 2009;338:b605.

11. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prog- nostic models in clinical practice. BMJ 2009;338:b606.

</p