Using the sociotechnical model to conduct a focused usability assessment of a breast reconstruction decision tool

Abstract

Introduction

BREASTChoice is a web-based breast reconstruction decision aid. The previous clinical trial—prior to the adaptation of this refined tool in which we explored usability—measured decision quality, quality of life, patient activation, shared decision making, and treatment choice. The current usability study was designed to elicit patients’ and clinicians’ perspectives on barriers and facilitators for implementing BREASTChoice into the clinical workflow.

Methods

We conducted qualitative interviews with patients and clinicians from two Midwestern medical specialty centers from August 2020 to April 2021. Interviews were first double coded until coders achieved a kappa > 0.8 and percent agreement > 95%, then were coded independently. We used a sociotechnical framework to evaluate BREASTChoice’s implementation and sustainability potential according to end-users, human-computer interaction, and contextual factors.

Results

Twelve clinicians and ten patients completed interviews. Using the sociotechnical framework we determined the following. People Using the Tool: Patients and clinicians agreed that BREASTChoice could help patients make more informed decisions about their reconstruction and prepare better for their first plastic surgery appointment. Workflow and Communications: They felt that BREASTChoice could improve communication and process if the patient could view the tool at home and/or in the waiting room. Clinicians suggested the information from BREASTChoice about patients’ risks and preferences be included in the patient’s chart or the clinician electronic health record (EHR) inbox for accessibility during the consultation. Human Computer Interface: Patients and clinicians stated that the tool contains helpful information, does not require much time for the patient to use, and efficiently fills gaps in knowledge. Although patients found the risk profile information helpful, they reported needing time to read and digest.

Conclusion

BREASTChoice was perceived as highly usable by patients and clinicians and has the potential for sustainability. Future research will implement and test the tool after integrating the stakeholder-suggested changes to its delivery process and content. It is critical to conduct usability assessments such as these prior to decision aid implementation to ensure success of the tool to improve risk communication.

Introduction

Clinical decision support (CDS) tools can facilitate point-of-care decision-making, particularly when they are thoughtfully designed to be user-centered and maximize principles of human-computer interaction [1, 2]. These types of tools, planned as both patient- and clinician-facing, have been successfully tested to improve a broad range of health outcomes [2, 3]. Breast reconstruction surgery restores the breast shape after mastectomy and can be performed at the time of mastectomy (immediate reconstruction) or months to years later (delayed reconstruction). Breast reconstruction can restore quality of life after mastectomy, but the risk of complications is relatively high. Many patients do not understand the risks and tradeoffs of the procedure, and decisions are often misaligned with patient preferences.

The previous clinical trial—prior to the adaptation of this refined tool in which we explored usability—measured decision quality, quality of life, patient activation, shared decision making, and treatment choice. In the prior study, the tool was tested as a website that patients logged into on their home computers or in clinic. Our previous work has demonstrated the efficacy of a CDS tool to support women’s decisions about post-mastectomy breast reconstruction (BREASTChoice) [4]. BREASTChoice is a web-based breast reconstruction decision aid that incorporates personalized risk estimates using data from the electronic health record (EHR), education about the pros and cons of breast reconstruction options, and a clinician summary to review at the point-of-care [5].

In response to feedback from the earlier trial and a stakeholder advisory board, preliminary work and tool adaptation included integrating photos, and improving the layout, flow of the risk page, and order of the information to ensure that they were patient-centered and relatable to users. A follow-up study evaluated factors that could impact implementation of the BREASTChoice tool according to patients, clinicians, and informatics professionals [4]. Stakeholders reported that BREASTChoice had the potential to facilitate shared decision-making, improve workflow, and enhance the efficiency of a breast reconstruction consultation. Prior to implementation of BREASTChoice in routine clinical care, stakeholders suggested exploring the function and use of particular features and factors which make the CDS tool conducive to use and sustainable. This study set out to test the usability of BREASTChoice in two settings with diverse patient populations.

Our overall objective is to implement the BREASTChoice tool in two academic medical centers for use among patients and clinicians. The current usability study was designed to elicit barriers and facilitators to ease the process of implementation and incorporate the tool into the clinical workflow from the perspective of patients and clinicians.

Methods

We conducted qualitative interviews with patients and clinicians from two Midwestern medical specialty centers and used a sociotechnical framework to evaluate BREASTChoice implementation and sustainability potential according to end-users, human-computer interaction, and contextual factors. The sociotechnical framework we employed has been used in our previous studies [6]. We hypothesized that we would identify modifiable factors to the workflow and content that would improve the tool’s utility and sustainability. All methods were carried out in accordance with relevant guidelines and regulations. All experimental protocols were approved by institutional review boards. Participants provided informed consent.

Conceptual framework

Using inductive thematic analysis, we adapted the sociotechnical framework (Fig. 1) as a guide to develop the codebook [6]. Framework constructs comprised the following: (1) people (participants, patients and clinicians); (2) workflow and communication (participant opinions regarding the timing of tool delivery and summary of tool content); (3) organizational policies and culture (participant perspectives on EHR integration of tool); (4) hardware and technical infrastructure (participant perceptions of utility of tablet, home, or clinic computers for delivery of the tool); (5) innovation content (participant views about tool content); (6) human-computer interaction (participant feedback regarding tool duration, and ease of navigating the tool); and (7) system monitoring and measurement (factors associated with sustained use of the tool).

Fig. 1

Adapted sociotechnical framework

Study population

Eligible clinicians included reconstructive surgeons, reconstructive surgery physician assistants, surgical oncologists, and patients of the reconstructive surgery department. Eligibility criteria for patients were English-speaking women over the age of 18 with a history of Stage I-III ductal or lobular carcinoma or ductal carcinoma in situ (DCIS), treated with mastectomy within the last five years were eligible to participate. Women who did not have mastectomy or had a diagnosis of a histology type besides ductal or lobular carcinoma or DCIS were excluded from participation. Women who had stage IV disease at the time of surgery or were being treated by institutions outside of the implementation sites were also excluded.

The rationale for recruiting women who had been through this decision-making process within recent years was to obtain their detailed perspective on the process while not adding burden to their healthcare and decision-making experience. We chose to group patients together regardless of type of reconstruction, since our study’s primary objective was to evaluate usability of the tool as opposed to the content of the tool. If a woman underwent two-stage implant reconstruction or tissue-expander surgery and then flap reconstruction, they were included in the immediate cohort. We chose not to include the names of the study sites for confidentiality purposes due to the relatively small number of clinicians employed at each site and thus eligible for inclusion.

Data collection

We created a semi-structured interview guide for patients and clinicians (see Appendix). We did not change the interview guide during the course of the study, but had asked our advisory board to review it prior to its use in the current study. We designed the interview questions to engage participants in a think-aloud format to get real-time feedback as they navigated through the website.

On the day of the interview, participants were sent a link to BREASTChoice, which was housed on a website outside the EHR. We explained the purpose of the study, and participants completed an informed consent or a waiver of informed consent. Interviews were conducted by masters-level research coordinators virtually and video recorded using Zoom between August 2020 and April 2021. Interviewers were trained and supervised by the principal investigators of the study (M.P., C.L.), both of whom have experience with qualitative interviewing and analysis. Interviews each lasted about 20–40 min, and field notes were taken during each session. We utilized the same interview guide with minor edits for it to make sense for both cohorts.

After the interview, participants completed a brief survey to assess demographic and professional (for clinicians and informatics experts) characteristics. Participants received $20 gift cards as remuneration for completing the interview and survey. Once the interviews were complete, the recordings were transcribed using a HIPAA-compliant transcription service and de-identified.

Coding/analysis

Results are reported in accordance with the sociotechnical framework. Transcripts were coded using QSR NVivo 12 using a codebook developed by research team members (C.P., K.C.). The two team members (C.P. and K.C.), were supervised by a principal investigator and senior team member (M.P. and R.F.). They double-coded two transcripts and checked for inter-rater reliability to ensure a kappa > 0.8 and percent agreement > 95%. They discussed discrepancies, revised the codebook as needed, and double-coded seven more transcripts. Once inter-rater reliability was obtained a second time, the remaining 13 transcripts were coded independently. Demographic and professional characteristics of participants were summarized using means and standard deviations for continuous variables and counts and percentages for categorical variables.

Results

Twenty-eight patients were approached, and ten (36%) were enrolled as we reached saturation. Seventeen clinicians were approached and twelve (71%) were enrolled. Table 1 displays the participant characteristics. Most patients (90%) and clinicians (67%) were white. Most patient participants had Stage I breast cancer (70%), and all patients had breast reconstruction after mastectomy. Six (50%) of the clinician participants were male, eight (67%) were physicians, and 54% had spent less than 10 years in practice.

Table 1 Participant Characteristics (N = 22)

Example quotes according to each sociotechnical dimension can be found in Table 2. Patient and clinician participants expressed that the existing tool could enhance communication during the encounter by providing background information on breast reconstruction choices and individualized risk. Patients and clinicians thought that having the patient use the tool at home or in the waiting room would prepare them for the breast reconstruction conversation during their surgical consultation. They felt that this timing would improve communication or clinical workflow. Some expressed concern that the time spent in the waiting room using the tool may not be sufficient for the patient to feel completely prepared for the conversation during the consultation.

Clinicians stated that they typically encounter CDS through a notification system embedded in the EHR. They felt that this strategy is not effective at engaging clinicians with patient-related information. Instead, clinicians would prefer the information to be included in the patient’s chart or the clinician’s EHR inbox prior to the consultation with the patient. Clinicians also expressed that it would be helpful to provide a tablet computer to patients when they are in the waiting room prior to their appointment so that they can complete the risk assessment and view the tool.

In terms of innovation content and human-computer interaction, patients and clinicians agreed that the tool contains helpful information, does not require much time for the patient to use, and fills gaps in knowledge in a methodical way. Tool navigation was acceptable to patients, although some expressed initial challenges in understanding how to progress through the tool or why the tool didn’t automatically update or repopulate when options were toggled on the risk assessment page. Patients also found the risk profile page helpful, yet a bit dense with information – both text and visuals – so that it took longer to read and digest.

Table 2 Example quotations according to sociotechnical dimensions, theme, and participant type

Discussion

We conducted qualitative interviews with patients and clinicians guided by a sociotechnical framework to evaluate BREASTChoice according to people, workflow and communication, organizational policies and culture, hardware and technical infrastructure, innovation content, and human-computer interaction. Consistent with the literature, which typically uses a minimum of 6 to 8 participants per cohort, we used a similar focused approach for our usability testing [1, 4].

Patients and clinicians thought that viewing the tool prior to the appointment would help the patient be better prepared for the breast reconstruction conversation during their surgical consultation. However, there may not be enough time to comprehensively review the tool in the waiting room, especially the risk profile page.

Additional modifiable factors to the workflow and content that would improve the tool’s utility and sustainability may include delivering the information via the EHR inbox to the clinician prior to the consultation with the patient, providing a tablet computer in the waiting room for patients to use to view the tool, making some small changes to the user interface for page navigation, and providing a clearer explanation or layout of the risk profile section of BREASTChoice.

Strengths of this study included its multicenter design to evaluate implementation and sustainability potential across sites. While the two sites were located in the Midwest, it is a strength of the study that usability was assessed and deemed acceptable among patients and clinicians in distinct practice settings with different workflows and patient characteristics. We additionally used a sociotechnical framework to guide the usability evaluation of BREASTChoice according to end-users (both patients and clinicians), human-computer interaction, and contextual factors. We also included MDs and physician assistants to represent the full scope of end-user clinicians. We evaluated risk communication, and improvements that could be made in terms of communicating risk, which remains one of the most challenging – and impactful – aspects of delivering appropriate decision aids via CDS.

Limitations included the relatively young group of clinicians and patients with limited racial or ethnic diversity. As a result, we acknowledge the potential for bias in the thematic results based on a lack of diversity of the participant cohorts. We could have had a more robust recruitment plan to reach more patients. In addition, we uncovered a lack of experience with usability testing among clinicians and patients, some difficulty among participants in interpreting open-ended interview questions, and clinician saturation with CDS (often referred to as “alert fatigue”). In addition, we did not evaluate the tool according to system monitoring and measurement, a component of the sociotechnical framework, since the tool had not yet been implemented and we were not seeking to evaluate how it impacted the technical ecosystem. However, this will be an important aspect of our ongoing evaluation once the tool goes live across sites.

Conclusions

This work demonstrated a high level of usability and potential for sustainability of BREASTChoice use among patients and clinicians. Our next step is to implement the tool across these two sites after integrating the suggested changes to workflow and content that we uncovered with this analysis. It is critical to conduct usability assessments such as these prior to CDS implementation to ensure success of the tool at the point-of-care.

Appendix

Summary table

What was already known on the topic.

• Clinical decision support (CDS) tools can facilitate point-of-care decision-making.

• CDS tools have been successfully tested to improve a broad range of health outcomes.

• Our previous work has demonstrated the efficacy of a CDS tool to support women’s decisions about post-mastectomy breast reconstruction (BREASTChoice).

• BREASTChoice has the potential to facilitate shared decision-making, improve workflow, and enhance the efficiency of a breast reconstruction consultation.

What this study adds to our knowledge.

• We demonstrate a high level of usability and potential for sustainability of BREASTChoice use among patients and clinicians.

• Our next step is to implement the tool across these two sites after integrating the suggested changes to workflow and content that we uncovered with this analysis.

• Usability assessments such as these are critical to conduct prior to CDS implementation to ensure success of the tool at the point-of-care.