Skip to main content
Download PDF

Application of a practice-based approach in variable selection for a prediction model development study of hospital-induced delirium

BMC Medical Informatics and Decision Making volume 23, Article number: 181 (2023) Cite this article

Abstract

Background

Prognostic models of hospital-induced delirium, that include potential predisposing and precipitating factors, may be used to identify vulnerable patients and inform the implementation of tailored preventive interventions. It is recommended that, in prediction model development studies, candidate predictors are selected on the basis of existing knowledge, including knowledge from clinical practice. The purpose of this article is to describe the process of identifying and operationalizing candidate predictors of hospital-induced delirium for application in a prediction model development study using a practice-based approach.

Methods

This study is part of a larger, retrospective cohort study that is developing prognostic models of hospital-induced delirium for medical-surgical older adult patients using structured data from administrative and electronic health records. First, we conducted a review of the literature to identify clinical concepts that had been used as candidate predictors in prognostic model development-and-validation studies of hospital-induced delirium. Then, we consulted a multidisciplinary task force of nine members who independently judged whether each clinical concept was associated with hospital-induced delirium. Finally, we mapped the clinical concepts to the administrative and electronic health records and operationalized our candidate predictors.

Results

In the review of 34 studies, we identified 504 unique clinical concepts. Two-thirds of the clinical concepts (337/504) were used as candidate predictors only once. The most common clinical concepts included age (31/34), sex (29/34), and alcohol use (22/34). 96% of the clinical concepts (484/504) were judged to be associated with the development of hospital-induced delirium by at least two members of the task force. All of the task force members agreed that 47 or 9% of the 504 clinical concepts were associated with hospital-induced delirium.

Conclusions

Heterogeneity among candidate predictors of hospital-induced delirium in the literature suggests a still evolving list of factors that contribute to the development of this complex phenomenon. We demonstrated a practice-based approach to variable selection for our model development study of hospital-induced delirium. Expert judgement of variables enabled us to categorize the variables based on the amount of agreement among the experts and plan for the development of different models, including an expert-model and data-driven model.

Peer Review reports

Background

Hospital-induced delirium among older adults has been estimated to cost the health care system in the United States from $38 to $152 billion every year [1]. Delirium is an acute neurocognitive syndrome and refers to a disturbance in attention and awareness with fluctuating intensity [2]. The sequelae of hospital-induced delirium include cognitive and functional decline [3]. Moreover, patients with hospital-induced delirium are at a two-times higher risk of death compared to patients without hospital-induced delirium [4].

At least 30% of hospital-induced delirium cases are preventable [5]. However, in order to prevent hospital-induced delirium, clinicians must be able to identify which patients are vulnerable and likely to develop delirium while they are hospitalized. Prognostic models that include potential predisposing and precipitating factors of hospital-induced delirium may be used to identify patients who are at risk of developing delirium and inform the implementation of tailored preventive interventions.

Many prognostic models of hospital-induced delirium exist specifically for older adults (Table S1 in the Supplementary Material) [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20] and often target patients who are hospitalized in intensive care units [10,11,12,13, 16, 18,19,20]. However, existing models for the older patient population vary in the ability to discriminate between patients with higher and lower probability of developing hospital-induced delirium [21]. The accuracy of these models is limited due to methodological shortcomings during model development and validation, including how candidate predictors and outcomes are operationalized [21].

Predictors are variables that are used to create a prognostic model [22]. Candidate predictors refer to all variables that are evaluated for their association with an outcome regardless of whether they are included in the final model [22]. It is recommended that candidate predictors are selected on the basis of existing knowledge [22]. This includes knowledge from the scientific literature as well as clinical expertise [23]. In our larger study, we are developing prognostic models of hospital-induced delirium for medical-surgical older adult patients using structured data from administrative and electronic health records. The purpose of this article specifically is to describe the process of identifying and operationalizing candidate predictors using a practice-based approach. This a necessary step towards transparent reporting of a prediction model study [24].

Methods

All of our methods were carried out keeping in mind the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) Statement. [24] Our methodological process included the following steps: (1) review of existing literature, (2) expertise-driven, manual extraction of variables that were used as candidate predictors in step 1, (3) expert judgment of variables that were extracted in step 2, (4) mapping of the variables from step 3 to the administrative and electronic health records, and development of operational definitions for the variables (operationalization of the variables).

Step 1: Review of existing literature

In our previous work, we conducted a systematic review of the literature to identify research designs and analytic methods that had been used to develop and validate prognostic model(s) of hospital-induced delirium for adult patients. The protocol for this review, including database-specific syntax, is available under registration number CRD42020218635 (version 03 December 2020) in the International Prospective Register of Systematic Reviews “PROSPERO”. We searched the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Medical Literature Analysis and Retrieval System Online (MEDLINE), American Psychological Association (APA) PsycInfo, and Web of Science Core Collection on August 22, 2020, to identify relevant studies. Details about the eligibility criteria and selection process are provided under “Additional Information about Step 1: Review of Existing Literature” in the Supplementary Material. A total of 42 studies were included, specifically 34 model development-and-validation studies and 8 validation-only studies (Figure S1 in the Supplementary Material).

Step 2: Expertise-driven, manual extraction of variables

The aforementioned review was used to extract variables. However, the inclusion criteria in the 03 December 2020 version (see “Eligibility Criteria” under “Additional Information about Step 1: Review of Existing Literature” in the Supplementary Material) were not specific to the medical-surgical older adult patient population, but included adult patients regardless of unit type. We still decided to use all of the 34 model-and-validation studies that were included in the original review for the variable extraction (Table S1 in the Supplementary Material). While we understand that risk factors of delirium may vary from patient population to patient population, there may also be some risk factors that are universal across healthcare settings. Furthermore, having screened 4,312 records for the review (Figure S1 in the Supplementary Material), we observed that the type(s) of unit were rarely specified in the articles. Because we did not want to leave out any potentially significant risk factors solely on the basis of limited information, we decided to keep our initial pool of variables broad and narrow it down in further steps of variable selection.

We referenced the “Candidate Predictors (Or Index Tests)” domain of the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies to check for the presence of relevant information about the variables that had been used as candidate predictors during model development (Table S2 in the Supplementary Material). [25] A researcher of Nursing Sciences with four years of experience as a registered nurse in the acute care setting (U.A.S.) manually extracted clinical concepts and their associated measurements (operational definitions). For example, both the concept “dehydration” and its measurement “blood urea to nitrogen ratio” were extracted from Carrasco et al.’s (2014) study [6] . Clinical concepts with missing measurements were not extracted, for example, “medical data”, “medication”, and “preoperative laboratory values”. [26].

Step 3: Expert judgment of variables

After we had identified and extracted the variables from the model development-and-validation studies, we created a list of these variables in a Microsoft Excel spreadsheet. In the spreadsheet, we programmed a dropdown with “Yes” and “No” answer options for each variable. We consulted experts from a multidisciplinary Iatrogenic Conditions Task Force (ICTF) (UF IRB #201900208) who judged whether each variable was associated with the development of hospital-induced delirium. The ICTF members included six registered nurses (three with a master degree, two with a bachelor degree, and one with a doctoral degree), two physical therapists (both with a doctoral degree), and one internal medicine physician.

We distributed the spreadsheet by e-mail and instructed the ICTF members to answer “Yes” or “No” to the question, “Do you consider this to be associated with delirium?”, for each variable. The exact wording of the e-mail is provided in the Supplementary Material (Box S1). The variables in the spreadsheet were ordered alphabetically not to potentially suggest that the variable above was more important than the variable below. We further encouraged the ICTF members to add their own ideas by including, “Would you like to add anything else? Please add in the column to the right.”, at the end of the spreadsheet.

Our goal was to generate true judgments from the ICTF members. We purposely did not allow for ambiguity in the survey by, for example, adding “Maybe” as an answer option. We already knew that the variables were “maybe” associated with hospital-induced delirium, because we had extracted them from the literature. In the case when an ICTF member selected neither “Yes” nor “No” for a variable, we assumed that the ICTF member had insufficient information to affirm that the variable was associated with hospital-induced delirium. Therefore, a blank answer was considered a “No”. The ICTF members were blinded to the source of the variables, i.e., that they had been extracted from the literature.

After each ICTF member had filled out the Excel spreadsheet, we counted the number of ICTF members who answered “Yes” and calculated the percentage of agreement for each variable. We then ranked the variables from complete agreement (9/9) to no agreement (0/9). The variables with complete agreement specifically were considered the “expert” candidate predictors.

Step 4: Mapping of variables to electronic health record system and development of operational definitions

We then explored whether and how each variable was represented in our local electronic health record (EHR) system, Epic, to be able to construct an operational definition of the variable for use in future studies that analyze data from the EHR system. A researcher of Nursing Sciences with four years of experience as a registered nurse in the acute care setting, specifically medical-surgical units in two different hospital systems (U.A.S.), in consultation with the ICTF members, reviewed all of the variables that had been extracted in step 2 and determined how each variable was represented in our Epic EHR system.

Reiterating the purpose of this study, we were only interested in the variables that could be operationalized using data that was (1) routinely collected in adult medical-surgical units, and (2) structured, such as coded data (for example, diagnosis codes), data entered in drop-down menus within thematic flowsheets, etc.

Results

We extracted 504 variables from 34 model development-and-validation studies. All of the studies are referenced and summarized in the Supplementary Material (Table S1). The complete list of the variables is also provided in the Supplementary Material (Table S3). Figure 1 presents how many times a variable was used as a candidate predictor across all of the studies. Two-thirds of the variables (337/504) were used as candidate predictors only once. One-third of the variables (167/504) were used as candidate predictors at least twice across all of the studies. The top three variables that were used as candidate predictors across at least half of the studies included age (31/34), sex (29/34), and alcohol use (22/34).

Fig. 1
figure 1

Appearance of Clinical Concept as Candidate predictor(s) across Prognostic Model Development-and-validation Studies of Hospital-induced Delirium. Note: This figure presents how many times a clinical concept was used as a candidate predictor across all of the prognostic model development-and-validation studies of hospital-induced delirium (n = 34). For example, 69 unique clinical concepts were used as candidate predictors twice.

Full size image

Table 1 presents the results of agreement among the ICTF members. Non-response was observed across 365 variables (72.4% of the 504 variables) and ranged from one ICTF member for 138 variables (37.8% of the 365 variables) to four ICTF members for one variable (0.27% of the 365 variables) with a median of two ICTF members. All of the ICTF members agreed that 47 variables were associated with hospital-induced delirium.

Table 1 Cumulative Number of Variables for Each Number of ICTF Members who Answered “Yes”
Full size table

Three variables with complete agreement (physical restraints, sedation status, and withdrawal) make up our operational definition of hospital-induced delirium that is the outcome in our larger model development study. These variables were excluded from our pool of candidate predictors to decrease bias in the outcome measure [22].

Other four variables with complete agreement were not considered further in this study, because they were exclusively captured as unstructured (text) data in our Epic EHR system. These variables were most accurately and reliably documented in clinicians’ narrative notes and included (1) duration of anesthesia, (2) physical status (measured with the American Society of Anesthesiologists Physical Status Classification), (3) severity of acute illness (measured with the Acute Physiology and Chronic Health Evaluation), and severity of stroke (measured with National Institutes of Health Stroke Scale).

Table 2 presents the list of the remaining 40 “expert” candidate predictors of hospital-induced delirium. Notably, 23 or 57.5% of them were included in the final prognostic models of hospital-induced delirium in the model development-and-validation studies based on statistical evaluation. These variables (empirical predictors) are denoted by asterisks in Table 2.

Table 2 “Expert” Candidate Predictors of Hospital-induced Delirium (n = 40)
Full size table

Based on how they were most accurately and reliably represented in our Epic EHR system, all of the variables that were extracted in step 2, including the variables that were later determined to be “expert” candidate predictors based on step 3, were divided into six operational categories, including administrative, diagnosis, laboratory test, medication, nursing assessment, and surgery (Table 2). Table 2 lists the “expert” candidate predictors only.

The “administrative” category includes variables that can be directly pulled from administrative databases (for example, age at encounter) or operationalized using multiple administrative data elements that have to be manipulated first (for example, the variable “length of hospital stay” is operationalized by subtracting the date of discharge from the date of admission).

The “diagnosis” category includes variables that can be operationalized using diagnosis codes based on the International Statistical Classification of Diseases and Health Related Problems, Clinical Modification (both the legacy ninth revision and the current tenth revision).

The “laboratory test” category includes values of laboratory tests that are coded using the Logical Observation Identifiers Names and Codes.

In addition to specific medications (for example, morphine sulfate), the “medication” category includes multiple medications that are grouped based on the classification that Epic uses for the Medication Administration Record.

The “nursing assessment” category includes variables that can be operationalized using (structured) data (that excludes free-text entries) from nursing flowsheets. Examples of nursing flowsheets that are routinely used by the medical-surgical nursing staff in our hospital system include the flowsheets under the “ADT Navigators” (“ADT” stands for “Admission, Discharge, Transfer”) and “Avatar” (flowsheets specific to airways, drains, lines, and wounds, so anything that is connected to, in, or on a patient), “Daily Care”, “Intake/Output”, “Pain Assessment”, “Simple Assessment”, and “Vital Signs Simple”. Various other flowsheets can be used for ad-hoc assessments (“Blood Transfusion”, “Restraints”, etc.).

The “surgery” category includes variables that can be operationalized using procedure codes based on the Current Procedural Terminology.

Discussion

We completed the first step of variable selection for the development of a prognostic model [23]. Based on the synthesis of data from the literature review, that resulted in the extraction of 504 unique variables that had previously been used as candidate predictors of hospital-induced delirium, and subsequent consultation with the multidisciplinary ICTF, we identified 484 clinical concepts that had been judged to be associated with the development of hospital-induced delirium by at least two ICTF members. Forty clinical concepts were determined to be “expert” candidate predictors of hospital-induced delirium based on the complete agreement among the nine ICTF members.

The most prominent finding is that we extracted over 500 unique candidate predictors of hospital-induced from the review of 34 separate studies. This number is conservative because we did not extract candidate predictors that had not been operationally defined in the articles. Such a heterogeneity among candidate predictors of hospital-induced delirium suggests a still evolving list of factors that contribute to the development of this complex phenomenon. The finding also supports the recommendations from domain experts to include clinical expertise in the selection of candidate predictors for prediction model development [23].

Consistent with these recommendations, we engaged the ICTF in our process of variable selection to triangulate the findings from our literature review and identify “clinically meaningful” candidate predictors of hospital-induced delirium. At least two ICTF members agreed on 96% of the variables that had been extracted from 34 different prognostic model development-and-validation studies of hospital-induced delirium. This finding suggests that close to all existing candidate predictors of hospital-induced delirium may be clinically meaningful. While these candidate predictors may be clinically meaningful, they may be, however, correlated with each other, and, therefore, redundant. In the second step of variable selection for the development of a prognostic model, statistical methods are used to determine which candidate predictors are important and which candidate predictors can be discarded without compromising the model performance [23]. If a number of candidate predictors is highly correlated with each other, only one candidate predictor may ultimately be selected for inclusion in the final prediction model.

Our process of variable selection has enabled us to categorize the variables based on the amount of agreement among the ICTF members and use these categorized variables in different models. In this study, our purpose was to identify candidate predictors that would be used for an expert-driven prognostic model of hospital-induced delirium. To feel confident in our future model, we decided to use candidate predictors that reached complete agreement among the nine ICTF members.

However, there is a limitation to this decision. The complete agreement among the ICTF members is the most stringent criterion for the inclusion of a candidate predictor in a model and one that risks the possibility of leaving out other variables that may be associated with the development of hospital-induced delirium. This possibility is high in our case, because, on average, two ICTF members did not provide a response to almost three-fourths of the variables in the survey. So, we are missing roughly two responses for most of our variables, and even up to four for one variable. Hypothetically, if we had not imputed a “No” wherever there was a missing response, and the two missing responses, on average, had been “Yes” responses instead, we could have identified additional 92 “expert” candidate predictors of hospital-induced delirium, because 139 clinical concepts would have then been judged to be associated with the development of hospital-induced delirium by all of the nine ICTF members (moving from 7/9 to 9/9 agreement in Table 2).

Considering this limitation, it is important to emphasize that we have not discarded the variables with less than 100% agreement among the ICTF members. They have been operationalized according to the process that we described earlier and will be used in different models. One of our plans is to apply machine learning to all of the variables that have been extracted from the literature and develop a data-driven model and compare it to the expert-driven model.

Once variables have been selected, they must be operationalized. However, most knowledge-driven variables describe abstract clinical concepts that do not directly map to the EHR data [27]. The engagement of ICTF members in our process of variable selection was necessary to understand how the variables that we had extracted from the literature were represented in our Epic EHR system. The ICTF members were also critical in helping us select the most accurate and reliable representation of a variable if the variable appeared in multiple places in the EHR. For example, instead of using a procedure code for transfusion of blood or blood components (Current Procedural Terminology code 36430), we were advised to use data from the ad-hoc nursing flowsheet “Blood Transfusion” that is initiated every time a transfusion of blood or blood components is administered.

We believe that involvement of clinical experts, not only in the selection of variables, but also in their operationalization, is an innovative approach across model development studies, because it takes into consideration current limitations of data within the EHR systems. These limitations, such as low accuracy of a variable that is operationalized with some EHR data element(s), may introduce bias to models that are based on the EHR data, unless these limitations are corrected. Furthermore, this approach is user-friendly and may be one, collaborative way to improve future uptake of models that will be implemented in clinical practice.

Lastly, we want to comment on the involvement of nurses, specifically, in our process of variable selection. Most of the ICTF members were nurses, and they participated in both the expert judgment of variables as well as the mapping of variables to the EHR system and development of operational definitions. Nurses’ expertise is unique because it comes from nursing surveillance that is a process of an ongoing and purposeful acquisition, interpretation, and synthesis of patient data [28]. Nursing surveillance requires nurses to be aware of any, even subtle, changes in patient status and make connections among patient characteristics and outcomes [28]. This truly practice-based knowledge may be what narrows the gap in the missing knowledge about the etiology of delirium and contributes to the discovery of new predisposing and precipitating risk factors of this, and other, iatrogenic conditions. To leverage nurses’ practice-based knowledge, we advocate for a greater use of specifically nurse-generated data (for example, nursing flowsheets) in the operationalization of candidate predictors, just as we proposed in this study, by creating the “nursing assessment” operational category. This is because nurses, in the process of nursing surveillance, collect in real or almost real time an abundance of data on a variety of characteristics that may be important for accurate prognosis of hospital-induced delirium or other iatrogenic conditions. For example, in a study that only used nurses’ assessment data for the development of a prognostic model of hospital-acquired pressure injuries, the best prognostic model had the area under the receiver operating curve of 0.80 and this indicates “excellent” discrimination [29, 30].

Limitations

There are a few important limitations to our study. First, nearly three out of four variables lacked, on average, two responses from an ICTF member. Because we categorized each missing response as a “No”, the relative effect of this decision may have implications for accurately identifying candidate predictors of hospital-induced delirium with any missing responses based on the amount agreement among the ICTF members. While this is an important limitation, it can be corrected by relaxing the criterion for the selection of candidate predictors, for example, using an 78% agreement instead of 100%. We did not do this, however, because we are planning to develop a separate model based on the candidate predictors that reached less than 100% agreement, so we are not discarding any potentially important candidate predictors of hospital-induced delirium. To avoid non-response altogether, a future study may consider designing a survey that would have “forced-entry” answer options, such as is possible in web-based applications for secure data capture.

Second, although we did try to diversify our task force by including members from two different hospitals, one in a rural and another in an urban location, the diversity of our task force may be considered to be limited, because it only included members from the same healthcare system. Clinicians that practice in the same healthcare system may be used to assessing risk of hospital-induced delirium in a different way than clinicians that practice in another healthcare system. A future improvement would be to capture experts’ past work experiences and/or to include experts from different healthcare systems to account for potential differences in the assessment.

Finally, we (temporarily) ignored four variables that were only documented in clinicians’ narrative notes. We also did not consider operationalizing other variables using text data. This is an important limitation, because 32% of clinical data is stored in the form of text, [31] and nurses’ narrative notes are particularly rich in potentially clinically meaningful information about the risk of iatrogenic conditions [32]. To address this limitation, we are conducting a separate study that uses natural language processing methods to identify risk factors of hospital-induced delirium in nurses’ narrative notes (see “Authors’ Information” below the article for more information about the study).

Conclusions

We described the process of identifying clinical concepts that can be used as candidate predictors to develop a prognostic model of hospital-induced delirium. While we focused on hospital-induced delirium, this process may also be applicable to other iatrogenic conditions. We highlighted the application of a practice-based approach during variable selection by engaging the Iatrogenic Conditions Task Force in the validation of clinical concepts that had been extracted from the literature. The empirical value of this approach needs to be tested in a future study that compares the performance of an expert-driven model against a data-driven model.

Data Availability

All of the data generated or analyzed during this study are included in the published manuscript and its supplementary material.

References

  1. Leslie DL, Marcantonio ER, Zhang Y, Leo-Summers L, Inouye SK. One-year health care costs associated with delirium in the elderly population. Arch Intern Med. 2008;168(1):27–32. https://0-doi-org.brum.beds.ac.uk/10.1001/archinternmed.2007.4.

    Article  PubMed  PubMed Central  Google Scholar 

  2. American Psychiatric Association. Neurocognitive disorders. In: Diagnostic and Statistical Manual of Mental Disorders. 5th ed., text rev. American Psychiatric Association;2022. Accessed January 24., 2023. https://0-doi-org.brum.beds.ac.uk/10.1176/appi.books.9780890425787.

  3. Inouye SK, Westendorp RG, Saczynski JS. Delirium in elderly people. Lancet. 2014;383(9920):911–22. https://0-doi-org.brum.beds.ac.uk/10.1016/S0140-6736(13)60688-1.

    Article  PubMed  Google Scholar 

  4. Aung Thein MZ, Pereira JV, Nitchingham A, Caplan GA. A call to action for delirium research: Meta-analysis and regression of delirium associated mortality. BMC Geriatr. 2020;20(1):325. https://0-doi-org.brum.beds.ac.uk/10.1186/s12877-020-01723-4. Published 2020 Sep 7.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Inouye SK, Bogardus ST Jr, Charpentier PA, et al. A multicomponent intervention to prevent delirium in hospitalized older patients. N Engl J Med. 1999;340(9):669–76. https://0-doi-org.brum.beds.ac.uk/10.1056/NEJM199903043400901.

    Article  CAS  PubMed  Google Scholar 

  6. Carrasco MP, Villarroel L, Andrade M, Calderón J, González M. Development and validation of a delirium predictive score in older people. Age Ageing. 2014;43(3):346–51. https://0-doi-org.brum.beds.ac.uk/10.1093/ageing/aft141.

    Article  PubMed  Google Scholar 

  7. de Wit HA, Winkens B, Mestres Gonzalvo C, et al. The development of an automated ward independent delirium risk prediction model. Int J Clin Pharm. 2016;38(4):915–23. https://0-doi-org.brum.beds.ac.uk/10.1007/s11096-016-0312-7.

    Article  PubMed  Google Scholar 

  8. Inouye SK, Viscoli CM, Horwitz RI, Hurst LD, Tinetti ME. A predictive model for delirium in hospitalized elderly medical patients based on admission characteristics. Ann Intern Med. 1993;119(6):474–81. https://0-doi-org.brum.beds.ac.uk/10.7326/0003-4819-119-6-199309150-00005.

    Article  CAS  PubMed  Google Scholar 

  9. Inouye SK, Charpentier PA. Precipitating factors for delirium in hospitalized elderly persons. Predictive model and interrelationship with baseline vulnerability. JAMA. 1996;275(11):852–7.

    Article  CAS  PubMed  Google Scholar 

  10. Ji MT, Xing SP, Yang Y. Pathophysiological factors of delirium among critically ill elders after non-cardiac surgery based on artificial neural networks: a pilot study. Anaesth Pain & Intensive care. 2018;22(4):424–30.

    Google Scholar 

  11. Jones RN, Marcantonio ER, Saczynski JS, et al. Preoperative cognitive performance dominates risk for delirium among older adults. J Geriatr Psychiatry Neurol. 2016;29(6):320–7. https://0-doi-org.brum.beds.ac.uk/10.1177/0891988716666380.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Kim MY, Park UJ, Kim HT, Cho WH. Delirium prediction based on hospital information (Delphi) in general surgery patients. Med (Baltim). 2016;95(12):e3072. https://0-doi-org.brum.beds.ac.uk/10.1097/MD.0000000000003072.

    Article  Google Scholar 

  13. Leung JM, Sands LP, Wang Y, et al. Apolipoprotein E e4 allele increases the risk of early postoperative delirium in older patients undergoing noncardiac surgery. Anesthesiology. 2007;107(3):406–11. https://0-doi-org.brum.beds.ac.uk/10.1097/01.anes.0000278905.07899.df.

    Article  CAS  PubMed  Google Scholar 

  14. Levkoff SE, Safran C, Cleary PD, Gallop J, Phillips RS. Identification of factors associated with the diagnosis of delirium in elderly hospitalized patients. J Am Geriatr Soc. 1988;36(12):1099–104. https://0-doi-org.brum.beds.ac.uk/10.1111/j.1532-5415.1988.tb04396.x.

    Article  CAS  PubMed  Google Scholar 

  15. O’Keeffe ST, Lavan JN. Predicting delirium in elderly patients: development and validation of a risk-stratification model. Age Ageing. 1996;25(4):317–21. https://0-doi-org.brum.beds.ac.uk/10.1093/ageing/25.4.317.

    Article  PubMed  Google Scholar 

  16. Pisani MA, Murphy TE, Van Ness PH, Araujo KL, Inouye SK. Characteristics associated with delirium in older patients in a medical intensive care unit. Arch Intern Med. 2007;167(15):1629–34. https://0-doi-org.brum.beds.ac.uk/10.1001/archinte.167.15.1629.

    Article  PubMed  Google Scholar 

  17. Pompei P, Foreman M, Rudberg MA, Inouye SK, Braund V, Cassel CK. Delirium in hospitalized older persons: outcomes and predictors. J Am Geriatr Soc. 1994;42(8):809–15. https://0-doi-org.brum.beds.ac.uk/10.1111/j.1532-5415.1994.tb06551.x.

    Article  CAS  PubMed  Google Scholar 

  18. Roijers JP, Rakké YS, Hopmans CJ, et al. Incidence and risk factors for delirium in elderly patients with critical limb ischaemia. Eur J Vasc Endovasc Surg. 2020;59(4):598–605. https://0-doi-org.brum.beds.ac.uk/10.1016/j.ejvs.2019.11.029.

    Article  PubMed  Google Scholar 

  19. Rudolph JL, Jones RN, Levkoff SE, et al. Derivation and validation of a preoperative prediction rule for delirium after cardiac surgery. Circulation. 2009;119(2):229–36. https://0-doi-org.brum.beds.ac.uk/10.1161/CIRCULATIONAHA.108.795260.

    Article  PubMed  Google Scholar 

  20. Zhan L, Wang XQ, Zhang LX. Nomogram model for predicting risk of postoperative delirium after deep brain stimulation surgery in patients older than 50 years with Parkinson disease. World Neurosurg. 2020;139:e127–35. https://0-doi-org.brum.beds.ac.uk/10.1016/j.wneu.2020.03.160.

    Article  PubMed  Google Scholar 

  21. Lindroth H, Bratzke L, Purvis S, et al. Systematic review of prediction models for delirium in the older adult inpatient. BMJ Open. 2018;8(4):e019223. https://0-doi-org.brum.beds.ac.uk/10.1136/bmjopen-2017-019223. Published 2018 Apr 28.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170(1):W1–W33. https://0-doi-org.brum.beds.ac.uk/10.7326/M18-1377.

    Article  PubMed  Google Scholar 

  23. Chowdhury MZI, Turin TC. Variable selection strategies and its importance in clinical prediction modelling. Fam Med Community Health. 2020;8(1):e000262. https://0-doi-org.brum.beds.ac.uk/10.1136/fmch-2019-000262. Published 2020 Feb 16.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Moons KG, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–W73. https://0-doi-org.brum.beds.ac.uk/10.7326/M14-0698.

    Article  PubMed  Google Scholar 

  25. Moons KG, de Groot JA, Bouwmeester W et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: The CHARMS Checklist. PLoS Med. 2014;11(10):e1001744. Published 2014 Oct 14. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pmed.1001744.

  26. Böhner H, Hummel TC, Habel U, et al. Predicting delirium after vascular surgery: a model based on pre- and intraoperative data. Ann Surg. 2003;238(1):149–56. https://0-doi-org.brum.beds.ac.uk/10.1097/01.sla.0000077920.38307.5f.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Sun J, Hu J, Luo D, et al. Combining knowledge and data driven insights for identifying risk factors using electronic health records. AMIA Annu Symp Proc. 2012;2012:901–10.

    PubMed  PubMed Central  Google Scholar 

  28. Kelly L, Vincent D. The dimensions of nursing surveillance: a concept analysis. J Adv Nurs. 2011;67(3):652–61. https://0-doi-org.brum.beds.ac.uk/10.1111/j.1365-2648.2010.05525.x.

    Article  PubMed  Google Scholar 

  29. Hosmer DW, Lemeshow S. Assessing the fit of the model. In: Hosmer DW, Lemeshow S, editors. Applied Logistic Regression. 2nd ed. John Wiley & Sons; 2000. pp. 143–202.

  30. Nakagami G, Yokota S, Kitamura A, et al. Supervised machine learning-based prediction for in-hospital pressure injury development using electronic health records: a retrospective observational cohort study in a university hospital in Japan. Int J Nurs Stud. 2021;119:103932. https://0-doi-org.brum.beds.ac.uk/10.1016/j.ijnurstu.2021.103932.

    Article  PubMed  Google Scholar 

  31. Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P. Availability of structured and unstructured clinical data for comparative effectiveness research and quality improvement: A multisite assessment. EGEMS (Wash DC). 2014;2(1):1079. Published 2014 Jul 11. https://0-doi-org.brum.beds.ac.uk/10.13063/2327-9214.1079.

  32. Bjarnadottir RI, Lucero RJ. What can we learn about fall risk factors from EHR nursing notes? A text mining study. EGEMS (Wash DC). 2018;6(1):21. https://doi.org/10.5334/egems.237. Published 2018 Sep 20.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the members and supporters of the Iatrogenic Conditions Task Force that has been formed as a result of the work for the UF-ECLIPSE study (University of Florida Electronic Health Record CLinical Data Infrastructure for Enhanced Patient Safety among the Elderly).

Funding

This study was supported by the National Institute on Aging, R33 AG062884; R.I.B. and R.J.L., Multiple Principal Investigators. The funding body played no role in the design of the study and collection, analysis, interpretation of the data, and in the writing of the manuscript.

Author information

Authors and Affiliations

  1. College of Nursing, Department of Family, Community, and Health Systems Science, University of Florida, 1225 Center Drive, PO Box 100197, Gainesville, FL, 32610, United States of America

    Urszula A. Snigurska, Laurence M. Solberg, Ragnhildur I. Bjarnadottir & Robert J. Lucero

  2. College of Public Health and Health Professions & College of Medicine, Department of Epidemiology, University of Florida, 2004 Mowry Rd, Gainesville, FL, 32610, United States of America

    Sarah E. Ser & Mattia Prosperi

  3. Geriatrics Research, Education, and Clinical Center (GRECC), North Florida/South Georgia Veterans Health System, 1601 SW Archer Rd, Gainesville, FL, 32608, United States of America

    Laurence M. Solberg

  4. College of Medicine, University of Central Florida, 6850 Lake Nona Blvd, Orlando, FL, 32827, United States of America

    Laurence M. Solberg

  5. Clinical and Translational Science Institute (CTSI), Integrated Data Repository Research Services, University of Florida, 3300 SW Williston Rd, Gainesville, FL, 32608, United States of America

    Tanja Magoc

  6. College of Medicine, Department of Health Outcomes & Biomedical Informatics, University of Florida, 2004 Mowry Rd, Gainesville, FL, 32610, United States of America

    Zhaoyi Chen & Jiang Bian

  7. School of Nursing, University of California Los Angeles, 700 Tiverton Ave, Los Angeles, CA, 90095, United States of America

    Robert J. Lucero

Authors
  1. Urszula A. Snigurska
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah E. Ser
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Laurence M. Solberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mattia Prosperi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tanja Magoc
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhaoyi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiang Bian
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ragnhildur I. Bjarnadottir
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Robert J. Lucero
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

R.J.L. conceptualized and designed this study. S.E.S. and U.A.S. prepared for the collection of data, collected the data, and analyzed the data. All of the authors U.A.S., S.E.S., L.M.S., M.P., T.M., Z.C., J.B., R.I.B., and R.J.L. participated in the interpretation of the results. U.A.S. drafted this manuscript. All of the authors U.A.S., S.E.S., L.M.S., M.P., T.M., Z.C., J.B., R.I.B., and R.J.L critically revised this manuscript for important intellectual content. All of the authors U.A.S., S.E.S., L.M.S., M.P., T.M., Z.C., J.B., R.I.B., and R.J.L have read and approved of the final version of this manuscript. Both R.J.L. and R.I.B. provided the funding.

Corresponding author

Correspondence to Urszula A. Snigurska.

Ethics declarations

Ethics approval and consent to participate

All of the procedures were performed in accordance with relevant guidelines and regulations as described in the protocol that was approved by the University of Florida Institutional Review Board (UF IRB #201900208). Informed consent was waived for all participants according to the United States federal law for secondary research (45 CFR). The findings reflect domain expertise of co-investigators covered under UF IRB #201900208.

Consent for publication

Not applicable.

Competing interests

M.P. is an Editorial Board Member of BMC Medical Informatics and Decision Making. All of the other authors declare that they have no competing interests.

Authors’ information (optional)

For more background information about this and the larger UF-ECLIPSE study, go to the National Institutes of Health RePORTER.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Snigurska, U.A., Ser, S.E., Solberg, L.M. et al. Application of a practice-based approach in variable selection for a prediction model development study of hospital-induced delirium. BMC Med Inform Decis Mak 23, 181 (2023). https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-023-02278-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-023-02278-1

Keywords

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us