Leveraging web-based prediction calculators to set patient expectations for elective spine surgery: a qualitative study to inform implementation

Abstract

Background

Prediction calculators can help set outcomes expectations following orthopaedic surgery, however effective implementation strategies for these tools are unknown. This study evaluated provider and patient perspectives on clinical implementation of web-based prediction calculators developed using national prospective spine surgery registry data from the Quality Outcomes Database.

Methods

We conducted semi-structured interviews in two health systems, Vanderbilt University Medical Center (VUMC) and Duke University Health System (DUHS) of orthopedic and neurosurgery health care providers (VUMC: n = 19; DUHS: n = 6), health care administrators (VUMC: n = 9; DUHS: n = 9), and patients undergoing elective spine surgery (VUMC: n = 16). Qualitative template analysis was used to analyze interview data, with a focus on end-user perspectives regarding clinical implementation of web-based prediction tools.

Results

Health care providers, administrators and patients overwhelmingly supported the use of the calculators to help set realistic expectations for surgical outcomes. Some clinicians had questions about the validity and applicability of the calculators in their patient population. A consensus was that the calculators needed seamless integration into clinical workflows, but there was little agreement on best methods for selecting which patients to complete the calculators, timing, and mode of completion. Many interviewees expressed concerns that calculator results could influence payers, or expose risk of liability. Few patients expressed concerns over additional survey burden if they understood that the information would directly inform their care.

Conclusions

Interviewees had a largely positive opinion of the calculators, believing they could aid in discussions about expectations for pain and functional recovery after spine surgery. No single implementation strategy is likely to be successful, and strategies vary, even within the same healthcare system. Patients should be well-informed of how responses will be used to deliver better care, and concerns over how the calculators could impact payment and liability should be addressed prior to use. Future research is necessary to determine whether use of calculators improves management and outcomes for people seeking a surgical consult for spine pain.

Introduction

Spine-related pain is among the largest drivers of disability and health care costs in the United States [1, 2]. It consistently ranks at or near the top in global reports of disease burden, and data suggest this burden is worsening. In 2016, US health care spending on spine pain was an estimated $134.5 billion (95% CI, $122.4-$146.9 billion), higher than spending attributed to any other health condition that year [1]. Surgery for spine pain is common, growing in incidence [3,4,5], and accounts for a significant proportion of spine-related health care costs.

Patient satisfaction has become an increasingly important metric by which patients and health care payers like Medicare and Medicaid judge the value of spine surgery [6,7,8]. Although most patients undergoing spine surgery experience significant benefits, recent work has found that up to 28% of patients are dissatisfied with their surgery despite achieving clinically relevant improvements in pain and function [9,10,11]. Some of the most important factors driving satisfaction after spine surgery include meeting expectations for return to work and return to previous physical activity [12,13,14,15]. Thus, one way to improve satisfaction with surgery is to ensure patients have realistic, evidence-informed post-surgical expectations during the decision-making process [16, 17].

Surgical prediction tools have been developed to guide pre-surgical counseling on patient-specific expectations for surgical outcomes [18]. These tools use patient-level characteristics to determine probable outcomes of surgery across domains such as adverse events (e.g. re-admission)[19, 20] and patient-centered outcomes (e.g. pain, disability) [21,22,23,24,25,26]. Recently, calculators were developed and internally validated using national data from the Quality Outcomes Database (QOD) [14, 22, 23, 27, 28]. These web-based tools provide individualized risk-adjusted postoperative projections for pain intensity, disability, quality of life, satisfaction, and return to work in patients undergoing elective lumbar and cervical spine surgeries.

Despite their potential value, health care providers have limited guidance on how to best implement prediction tools to facilitate pre-surgical counseling. Therefore, the objective of this study was to conduct interviews with health care providers, administrators, information technology (IT) professionals, and patients to assess barriers, opportunities, and optimal strategies for prediction tool implementation. In particular, we were interested in evaluating topics related to user characteristics, calculator content and interface, workflow integration, organizational culture, and external regulations that would inform implementation strategies for the QOD calculators as well as other prediction tools. We used the Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems [29] as a framework to inform interview content. This 8-dimensional model is specifically designed to address the socio-technical challenges involved in design, development, implementation, use, and evaluation of health IT within complex adaptive healthcare systems. We focused on QOD calculators predicting pain and disability for this project as these outcomes are often of highest importance to patients undergoing spine surgery [9, 30,31,32], and we expect similar implementation strategies to apply to use of other comparable calculators. We conducted interviews in two health care systems, Vanderbilt University Medical Center (VUMC) and Duke University Health System (DUHS), to compare implementation needs in systems that were familiar and unfamiliar, respectively, with the QOD calculators.

Methods

QOD predictive calculators

For this study, we focused on the QOD web-based calculators that predict disability and pain intensity (back/neck and leg/arm) following lumbar and cervical spine surgery. The calculators provide probabilities for (1) any improvement over current level of pain or disability, and (2) an improvement of 30%, which is a valid criterion for minimally clinical important difference (MCID) [22, 23]. Patients enter information about their demographics, symptoms, and pain and disability level into the calculator via a web-based interface. This information is then used to predict each outcome probability. Additional details on calculator content and development are provided elsewhere [22, 23, 33].

Participant selection

We used purposive sampling to recruit stakeholders located at VUMC and DUHS that met one of the following criteria: (1) surgeon involved in the care of individuals undergoing spine surgery; (2) non-surgeon health care provider involved in the care of individuals undergoing or potentially eligible for spine surgery; (3) health system administrator or health IT faculty or staff involved in the clinical spine care process; (4) patient with low back and/or neck pain consulting with a surgeon. We focused purposive sampling on ensuring a representative sample based on role (for both health care system interviewees) and experience with the calculators (for VUMC health care system interviewees). Purposive sampling of patients focused on gathering a representative sample of interviewees based upon the presence of neck versus back pain. We selected VUMC because familiarity of their health care providers with the calculators allowed us to explore their depth and breadth of experience, with key insights into potential implementation barriers of these specific tools. On the other hand, DUHS interviewees would provide important perspectives on potential barriers and facilitators for clinical sites implementing an entirely novel tool. Institutional Review Board approval was obtained prior to study activities at both sites. All interviewees provided informed consent before participation.

Data collection and management

Interviews (one-on-one) took place between April and December 2021. A PhD-trained investigator with 2 years of qualitative interview experience (TL, male) conducted all interviews by video call (i.e., Zoom). Provider and administrator/health IT interviews began with a short demographic survey (position/job title, stakeholder group, department, years in practice and at the institution, gender, race, ethnicity, and age). The patient demographic survey included age, race, ethnicity, employment status, level of education, primary diagnosis, acuity of symptoms, history of spine surgery, smoking status and self-reported health status compared to others. The interviewer then described the QOD calculators and showed screenshots of the data entry interface, as well as example outputs from mock data entered into the calculator (Additional File 1). The interviewer used the example outputs to describe how the outputs (i.e., the probabilities of achieving any improvement over current level of pain or disability, and an improvement of 30% over current level of pain or disability) would aid with setting expectations prior to surgery. After this demonstration, the interviewer provided opportunities for the interviewee to ask questions. The interviewer then followed a semi-structured interview guide, informed by the Socio-technical model and customized for each type of interviewee (Additional File 2 and 3). When discussing options for how the calculator should display probability outputs, we tried to minimize bias by showing the same output options to each interviewee, using a standard script, and asking standardized questions as outlined in the semi-structured interview guide. Interviews lasted 30–45 min and were audio recorded, transcribed verbatim, and assessed for data quality before analysis. Key topics covered by semi-structured interviews for each stakeholder type are provided in Table 1.

Table 1 Interview Topics^a

Data analysis. We used a qualitative template analysis approach [34], a form of thematic analysis emphasizing hierarchical coding. Dedoose software Version 9.0.18 was used to organize and analyze interview data. Two team members (TL, BR) read each interview for familiarity. The initial coding template consisted of interview guide prompts and expected categories of responses. TL and BR coded each transcript, and, in an iterative fashion, met to compare codes and reach consensus on coding application after each set of 4–5 interview transcripts were analyzed. Saturation of codes, meaning no new codes or code categories were found in subsequent interviews, occurred after 15 interviews (5 patients; 5 administrators; 5 health care providers) and coders agreed on the analytic template to use for coding subsequently scheduled interviews. We determined that data saturation was achieved with the number of interviews we completed (24 health care provider interviews, 18 administrator/health IT interviews, and 16 patient interviews). We did not plan to conduct a formal comparative analysis of results from VUMC and DUHS interviews, but rather identify where potential implementation challenges converged or diverged between sites that were familiar and unfamiliar with the calculators, respectively.

Results

Health care provider stakeholders included n = 18 (10 surgeons, 8 non-surgeons) at VUMC and n = 6 (5 surgeons, 1 non-surgeon) at DUHS (Table 2). Administrator/health IT stakeholders included n = 9 at VUMC and n = 9 at DUHS. Patient stakeholders (n = 16) were from VUMC only, with 10 having neck pain and 6 with low back pain. Table 3 provides additional patient demographic and health related information. Tables 4 and 5 provide sample quotes from interviewees. Table 6 outlines actionable strategies to optimize implementation of the QOD calculators in clinical settings based on our holistic synthesis.

Table 2 Demographic information for health care provider participants

Table 3 Demographic and health related information for patient participants

Table 4 Sample health care provider, administrator, and health IT interviewee quotes by topic

Table 5 Sample patient interviewee quotes by topic

Characteristics of the user

Perceived utility of using an outcomes calculator by providers

Almost all administrators had a favorable impression of the calculators and felt implementation would vary by surgeon depending on whether they found it useful for patient education. Across surgeons, the most commonly stated potential benefits were as a resource to dissuade patients from wanting surgery if they were poor surgical candidates (i.e., no correctable pathology) and to set realistic outcomes expectations. Most surgeons felt the calculators would result in outcome probabilities that were generally similar to what they could surmise through more traditional methods (i.e., talking to the patient, considering imaging results). For this reason, a few surgeons felt the calculators would be of limited use. However, the majority of surgeons reported that having specific, evidence-based probability estimates would help reinforce their clinical impressions while setting appropriate surgical expectations for patients. Non-surgeon health care providers shared these perspectives on the potential benefits. DUHS stakeholders were largely unfamiliar with the QOD calculators and expressed more skepticism about the added benefit of these calculators and concerns over validity of the outcome probability estimates than those at VUMC.

What is important to patients

We solicited input from patients on what was most important to them when deciding whether to undergo surgery. The most common responses were the probability of returning to work or leisure activities, improvement in pain and function, and symptom severity. Each patient we interviewed had a generally favorable impression of the calculators. Most reported that the greatest value was having quantitative, evidence-informed estimates of outcomes probabilities that would reinforce expectations set by their surgeon. However, some patients stated the calculator would be more helpful if it compared outcomes probabilities for different surgical and non-surgical options.

Clinical content

Quality and applicability of the calculators

The QOD calculators was developed and tested in patients undergoing elective spine surgery for degenerative conditions. A common concern from surgeons was for how well the QOD sample represented their own patient population. These concerns were specific to surgeons that performed complex, or minimally-invasive, spine procedures that were not well-represented in the QOD. Some surgeons felt the calculators did not include all potentially-relevant predictors, like current opioid use. Others felt the utility of the calculators was limited because they do not provide probabilities of improvement with other treatments, and therefore would be unhelpful for setting expectations for surgical versus non-surgical outcomes.

Redundancy of information

A significant concern across health care providers and administrators was the increased response burden associated with adding questions to the existing battery of questionnaires patients already complete in conjunction with visits. Interestingly, additional response burden was not a major concern for most patients interviewed. While they stated that measures should not be redundant, they were largely in favor of completing calculator items. Patients reported that they would be more likely to complete additional questions if they understood how surgeons would use the information to inform their care.

Workflow and communication

The biggest concern for providers and administrators was interrupted clinical workflow due to use of the calculators. Interviewees had different perspectives on how to mitigate this concern by strategically selecting who should complete the calculator and when they should complete it.

Selecting who should complete the calculators

Across health care providers, there was strong consensus that only patients who are already determined to be a surgical candidate should complete the calculators. Providers felt it would otherwise lead to wasted effort, and potentially higher patient demand for unwarranted surgeries. However, many administrators and IT professionals felt that taking a standardized approach to administering the tools to all patients would optimize efficiencies and enhance implementation potential. These stakeholders suggested that patients would only see their results if surgery was deemed suitable.

Determining when to complete the calculator

We also observed a wide range of opinions on when and how calculators should be administered. Those advocating for universal administration supported completing the calculator at home prior to the initial visit. Links to the calculator would be sent through MyChart or MyHealth interfaces or through email. Clinical staff liked this option because frequent software and hardware challenges (e.g., connectivity issues, depleted batteries) make use of clinic tablets unreliable. Moreover, clinic staff reported that patients would commonly need assistance navigating the tablet interface, further slowing workflow. Most patients also preferred the at-home option because they could complete questionnaires “on their own time, on their own device” and “not when rushed.”

Those advocating for a more targeted approach to administration felt patients should complete the calculator after being identified as a surgical candidate, but before making a decision to undergo surgery. Often, this interval occurs while the patient is in clinic, which introduces the challenge of completing a calculator while minimizing disruption to clinical workflows. One suggestion was to have patients complete the calculator after being triaged as a potential surgical candidate by an advanced practice provider (APP). In most circumstances, the APP and surgeon follow-up visits occur on different days, allowing for an interval in which patients could complete the calculator at home. Another option was to have patients complete the calculators after the visit with their surgeon, but before scheduling surgery. This way, any concerns regarding outcomes could be discussed with the surgeon before scheduling.

A subset of surgeons and non-surgeons expressed strong preferences for completion of the calculator in clinic. They felt patients should only see calculator results while in the presence of a health care provider. The concern was that patients could misinterpret the findings or fail to comprehend how probabilities should inform care decisions. Moreover, some providers had concerns about the quality of data entered into the calculators, noting they would only trust calculator outcomes if they knew patients understood the questions, completed the questions themselves, and were not trying to “game the system” by entering inaccurate responses. Providers felt such concerns were mitigated by having patients complete the calculators in clinic. Regardless of the specific approach, administrators suggested a pilot of the calculators with one or more clinical champions. A pilot would enable health care systems to resolve workflow bottlenecks, identify optimal windows and formats for administration, gather feedback from end-users, and establish IT requirements.

Human/Computer interface

Communicating probabilities and key terms

Interviewees were presented with multiple options for visualizing results. A consensus among all patients and providers was that probabilities should be presented clearly, with explanations of key terms like “disability” and “improvement”. Patients preferred to see their current level of pain or disability and what a 30% improvement would be for each of those outcomes. Another consensus among patients was that results should not be presented using statistical graphs or figures, but in more simplistic forms with whole numbers and concise, layperson descriptions of the outcomes. While patients overwhelmingly supported the potential value of the calculators, some believed the benefit was limited, wanting to know probabilities for higher improvement thresholds like 50% improvement (“30% improvement would not be enough for me”), probability of no pain, or comparison of outcome probabilities across various treatments. Nearly all patients preferred to have a surgeon or other health care provider on hand to discuss results in detail.

Resource needs and constraints

Most surgeons stated they were comfortable discussing outcomes probabilities with patients. Nevertheless, some stated a script or talking points specific to capabilities and limitations of the calculators could be helpful. Regardless of which patients completed the calculators and when, nearly all providers and administrators agreed that at least some staff resources would be necessary for implementation (e.g., ensure patients have completed the calculator, be on hand to answer questions, remind surgeons to discuss calculator results).

Since the calculators represent new tools for surgeons, we were interested in processes that would help them remember to incorporate results into patient counseling discussions. There was no consensus on the best way to deliver reminders. Some surgeons and their staff preferred pop-up reminders in the electronic health record (EHR), while others were concerned for notification fatigue. A few suggested building reminders into the clinic notes, but qualified that these passive reminders alone would not be sufficient. Few saw benefit in placing flyers around the clinical workspace. The majority of providers felt support staff should be leveraged to remind providers, but that the optimal strategy would vary across clinics and surgeons. One consistent suggestion was that surgeons be reminded of the calculators during regular staff meetings.

Internal organizational policies, procedures, and culture

Culture of research and innovation

Interviewees at both VUMC and DUHS felt the strong research culture and emphasis on quality care in both systems created an environment that would support implementation. Many interviewees stated that practice variability and siloed care delivery across clinics would largely preclude a single institutional implementation strategy. Most administrators and non-surgeon health care providers felt that gaining buy-in from all surgeons would be nearly impossible because some would not see value in the calculators. This observation was supported by skepticism from some surgeons who, as previously mentioned, felt these tools might not apply to their patient population or include the appropriate risk factors. A common opinion across interviewees was that successful implementation would require strong buy-in from Department and Division chiefs.

External rules, regulations, and pressures

Many providers and administrators expressed concerns that payers could use calculator results to deny coverage. These concerns led some to recommend against documentation of results in the EHR and to suggest that health care systems are clear about which entities will have access to results. Some interviewees raised concerns over whether use of a calculator could expose providers and health care systems to liability if outcomes did not align with calculator results (e.g., continued pain and disability despite high probabilities of success). To address liability concerns, several providers suggested a “warning label” on the calculators and/or confidence intervals for the probability estimates. These additions would ensure patients understood the uncertainty around probability estimates and that results should be used solely as an adjunct to other information when making treatment decisions. Most patients understood the inherent uncertainty in predicting surgical outcomes, but agreed that surgeons should be transparent in discussing calculator strengths and limitations.

System Measurement and Monitoring

Implementing a new clinical tool consumes time and resources, therefore we were interested in learning how to measure the potential benefits of these efforts. We asked interviewees how they would determine if use of the tools to set outcomes expectations were a “success” in practice. Common metrics noted by health care providers and administrators were improvement in patient satisfaction scores and general improvements in patient management or decision-making from the provider perspective. Health care providers especially valued a reduction in decisional conflict among their patients who may be struggling with the decision to undergo surgery. Patients also reported the potential benefits of reducing decisional conflict, while emphasizing the most important measure of success was the degree to which the calculators accurately predicted their outcomes. Approximately half of the patients stated that improved quality of the patient-provider interaction should be another quality metric by which to evaluate the benefit of the calculators.

Discussion

This work aims to inform implementation strategies for tools that support patient counseling on expectations and addresses the tension between additional automated tool application and the need for better ways to facilitate shared-decision-making. One strength of this qualitative analysis is the consideration of viewpoints across various stakeholder types at two institutions. We found common potential challenges across institutions, including the need for leadership buy-in, difficulties integrating a web-based tool into existing IT workflows, process variations across clinics that limit integration, and concerns over data quality, liability, and impact on payment. Prior implementation studies of comparable tools in different clinical settings have reported similar barriers [35,36,37,38]. While most interviewees agreed the calculators would help to set surgical expectations, they reported widely divergent viewpoints on the best way to implement these tools. These findings suggest that no single implementation strategy would be universally successful, even within the same healthcare system. Nevertheless, actionable strategies emerged throughout our interviews to guide implementation (Table 6). We believe, given the study design, that these determinations have broad capacity for translation in other clinical contexts that are similar to our own.

Table 6 Actionable strategies to optimize successful implementation of the QOD calculators

This study explored perspectives across stakeholders in two distinct health care systems that differed in their familiarity with the calculators. Notable similarities across systems include the need for department or division chief buy-in, challenges with integrating a web-based tool into existing EHR and documentation workflows, variation across clinics in workflows and processes that limit integration, and concerns over liability and impact on payment. As with VUMC interviewees, the majority of DUHS interviewees felt that a single implementation strategy was not feasible, but should be tailored to the unique needs, resources, and settings of each clinic and/or surgeon.

Although implementation challenges were similar across systems, the major differences were that DUHS interviewees (1) expressed more skepticism about the validity or accuracy of the calculators and (2) had more doubts about the operational feasibility of assimilating them into care. Most DUHS interviewees stated they would need more familiarity with the calculators before considering their use, and evidence of their benefit in setting expectations beyond current practice. Compared to health care providers, patients more readily identified the value of having access to quantifiable, evidence-based outcomes probabilities. The difference in value judgements between patients and providers on this point highlights an important opportunity to enhance patient-provider communication for surgical outcomes expectations. The majority of patients interviewed liked to see evidence-based estimates that supported their surgeons’ recommendations and felt this information would further reinforce their surgical decisions. Health care providers may not recognize or fully appreciate this patient perspective on the calculators’ value.

Our findings suggest a few critical implementation strategies. These include articulating the accuracy and validity of the calculators to providers, demonstrating their operational feasibility, and convincing providers of the potential value they bring in contributing measurable, evidence-based outcomes probabilities to the clinical encounter. Implementation strategies should also address concerns over who would have access to calculator results and how they could influence payment and liability, as interviewees at both sites raised these issues. Such efforts may include piloting the calculators with one or more clinical champions to provide “proof of concept” regarding benefits, a recommendation raised by a few administrators.

Many providers and administrators expressed doubt that patients would be willing to complete additional questions associated with the calculators. On the contrary, most patients were willing to complete the calculators as long as questions were not redundant and providers made it clear how the information would direct care. This feedback has two main implications. First, it suggests the need to limit redundancy of data entry by using information from existing data sources to populate calculator responses when possible (e.g., importing demographic or diagnostic data elements from the EHR). Second, this feedback reinforces the importance of clearly communicating the intent of clinical data collection to improve response rates and quality of entered data.

Perhaps the most important perceived driver of implementation success is the ease in which patients and providers can use the calculators during routine clinical care. Given the inherent variability in clinical workflows, successful implementation of the calculators at scale would require flexibility on who completes them, and when and how they are completed based on clinic resources, surgeon preferences, and patient needs. Strategies to enhance ease of use for patients includes providing flexible options for them to complete the calculator at-home or in-clinic, and having readily available explanations of key terms (e.g. disability, probability, and improvement). For providers, ease of use would be enhanced by integrating the calculators into the EHR, leveraging clinical support staff to remind surgeons to administer/discuss calculators with patients, receiving reminders about the calculators at staff meetings, and developing simple talking points or scripts to facilitate discussion with patients. The perceived optimal timing of when surgeons discuss probabilities with patients was highly variable and a very small proportion stated they were not interested in outcomes probabilities at all. Thus, health care providers should talk with patients about this resource and their preferences for its use in discussions about surgery.

This work has some limitations. First, we conducted interviews in two academic health systems. The characteristics of these systems and interviewees may not be generalizable to all healthcare settings. Additionally, this study evaluated the implementation potential of specific calculators. While most calculators share similar attributes, we do not know whether the challenges and implementation strategies we identified would apply to other calculators currently available, such as calculators that predict return to work [26].

A second limitation is that our sample of patients had limited diversity in race and ethnicity. We only recruited patient participants from one center. Therefore, the views of these patients may not generalize to more diverse populations in other clinical contexts. Our sample was similar in level of symptom acuity to studies on spine surgery populations using QOD data [14, 22, 23, 27, 28], however tended to be older (median age = 60 years, mean age = 62.2 years) compared to QOD studies (mean ranges 51–55 years). Our sample also had higher levels of education, a higher proportion of male participants, and was under-represented by non-White and Hispanic participants compared to published QOD studies. These comparisons are important because they suggest our study did not fully capture perspectives of younger, non-White, and Hispanic participants undergoing spine surgery, or those with lower educational attainment. These patients may have different communication, informational, or access needs. They may see different value in the calculators, have differing perspectives on the burden of extra questions, want information earlier or later in the care episode, or value information on other outcomes. Studying needs specific to these under-represented populations would further inform calculator implementation for an important care-seeking group. Finally, sample recruitment was driven by a desire to develop a representative cohort of interviewees, not necessarily by the desire for theme saturation. Although we did see similar and consistent responses across interviewees with similar roles, a larger sample may have uncovered additional perspectives.

Our findings have important implications for future work on the topic. Most notably, the calculators studied provide outcomes probabilities as the result of surgery, however are unable to provide outcomes probabilities for other non-surgical treatment options. Multiple patient interviewees noted the importance of comparing outcomes probabilities among different treatment options. To our knowledge, no single resource exists that directly compares personalized outcomes probabilities of different treatments for spine pain. Future work should focus on developing these resources, as well as calculators that predict longer-term outcomes.

Second, future research should focus on understanding how implementation strategies may differ across diverse healthcare settings and populations. This would include determining optimal strategies for communicating key terms (e.g., outcomes probabilities) to patients with differing levels of education and health literacy. It would also entail determining best ways to educate patients and providers on the limitations of the calculators, i.e., that they cannot be used to compare outcomes probabilities for different treatments. Our work also lays a foundation for future effectiveness studies. Specifically, it outlines strategies to ensure successful implementation of these tools and optimize fidelity to protocols that use these tools in pragmatic trials. Such trials are necessary to determine whether use of calculators improves management and outcomes for people seeking a surgical consult for spine pain.

A final direction for future work is to ensure that prediction tools, such as surgical outcomes calculators, do not lead to inequitable care by creating bias in clinical decision-making. This is a major concern as health care systems leverage new technologies and large, population-based datasets to create decision-support algorithms [39,40,41]. Some surgeons in this study raised concerns that the calculators may not apply to their specific patient population. This concern highlights an important principle; that outcomes probabilities will be most accurate when surgeons use calculators in a patient population similar to the one used to develop the calculator. Any future calculator refinement for more specific populations should pay careful attention to design and take a thoughtful approach to mitigate any potential biases.

Conclusions

Interviewees had a largely positive opinion of the QOD calculators, believing they would aid in setting appropriate expectations for spine surgery outcomes. Implementation strategies will need to vary according to clinic resources, surgeon preferences, and patient needs. The most successful strategies will provide flexible options for where and how patients complete the calculator, using existing EHR data elements when available to minimize response burden. To encourage calculator use by surgeons, implementation strategies should leverage clinical support staff to assist with administration, include reminders during regular staff meetings, provide scripts or talking points to aid in patient education on outcome probabilities, and address any concerns over data quality, liability, and impact on payment. Future research is necessary to determine whether use of calculators improves management and outcomes for people seeking a surgical consult for spine pain.