Analysis of clinical decision support system malfunctions

A case series and survey

Adam Wright, Thu Trang T Hickman, Dustin McEvoy, Skye Aaron, Angela Ai, Jan Marie Andersen, Salman Hussain, Rachel Ramoni, Julie Fiskio, Dean F. Sittig, David W. Bates

Research output: Contribution to journalArticle

Abstract

Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually.Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate.Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

Original languageEnglish (US)
Article numberocw005
Pages (from-to)1068-1076
Number of pages9
JournalJournal of the American Medical Informatics Association
Volume23
Issue number6
DOIs
StatePublished - Nov 1 2016

Fingerprint

Clinical Decision Support Systems
Software
Amiodarone
Electronic Health Records
Platelet Aggregation Inhibitors
Surveys and Questionnaires
Aspirin
Thyroid Gland

Keywords

  • Anomaly detection
  • Clinical decision support
  • Electronic health records
  • Machine learning
  • Safety

ASJC Scopus subject areas

  • Health Informatics

Cite this

Wright, A., Hickman, T. T. T., McEvoy, D., Aaron, S., Ai, A., Andersen, J. M., ... Bates, D. W. (2016). Analysis of clinical decision support system malfunctions: A case series and survey. Journal of the American Medical Informatics Association, 23(6), 1068-1076. [ocw005]. https://doi.org/10.1093/jamia/ocw005

Analysis of clinical decision support system malfunctions : A case series and survey. / Wright, Adam; Hickman, Thu Trang T; McEvoy, Dustin; Aaron, Skye; Ai, Angela; Andersen, Jan Marie; Hussain, Salman; Ramoni, Rachel; Fiskio, Julie; Sittig, Dean F.; Bates, David W.

In: Journal of the American Medical Informatics Association, Vol. 23, No. 6, ocw005, 01.11.2016, p. 1068-1076.

Research output: Contribution to journalArticle

Wright, A, Hickman, TTT, McEvoy, D, Aaron, S, Ai, A, Andersen, JM, Hussain, S, Ramoni, R, Fiskio, J, Sittig, DF & Bates, DW 2016, 'Analysis of clinical decision support system malfunctions: A case series and survey', Journal of the American Medical Informatics Association, vol. 23, no. 6, ocw005, pp. 1068-1076. https://doi.org/10.1093/jamia/ocw005
Wright, Adam ; Hickman, Thu Trang T ; McEvoy, Dustin ; Aaron, Skye ; Ai, Angela ; Andersen, Jan Marie ; Hussain, Salman ; Ramoni, Rachel ; Fiskio, Julie ; Sittig, Dean F. ; Bates, David W. / Analysis of clinical decision support system malfunctions : A case series and survey. In: Journal of the American Medical Informatics Association. 2016 ; Vol. 23, No. 6. pp. 1068-1076.
@article{7a69a3b63a064bdcba97e8a2b3e77668,
title = "Analysis of clinical decision support system malfunctions: A case series and survey",
abstract = "Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93{\%} of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually.Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate.Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.",
keywords = "Anomaly detection, Clinical decision support, Electronic health records, Machine learning, Safety",
author = "Adam Wright and Hickman, {Thu Trang T} and Dustin McEvoy and Skye Aaron and Angela Ai and Andersen, {Jan Marie} and Salman Hussain and Rachel Ramoni and Julie Fiskio and Sittig, {Dean F.} and Bates, {David W.}",
year = "2016",
month = "11",
day = "1",
doi = "10.1093/jamia/ocw005",
language = "English (US)",
volume = "23",
pages = "1068--1076",
journal = "Journal of the American Medical Informatics Association : JAMIA",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "6",

}

TY - JOUR

T1 - Analysis of clinical decision support system malfunctions

T2 - A case series and survey

AU - Wright, Adam

AU - Hickman, Thu Trang T

AU - McEvoy, Dustin

AU - Aaron, Skye

AU - Ai, Angela

AU - Andersen, Jan Marie

AU - Hussain, Salman

AU - Ramoni, Rachel

AU - Fiskio, Julie

AU - Sittig, Dean F.

AU - Bates, David W.

PY - 2016/11/1

Y1 - 2016/11/1

N2 - Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually.Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate.Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

AB - Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually.Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate.Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

KW - Anomaly detection

KW - Clinical decision support

KW - Electronic health records

KW - Machine learning

KW - Safety

UR - http://www.scopus.com/inward/record.url?scp=84994783028&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84994783028&partnerID=8YFLogxK

U2 - 10.1093/jamia/ocw005

DO - 10.1093/jamia/ocw005

M3 - Article

VL - 23

SP - 1068

EP - 1076

JO - Journal of the American Medical Informatics Association : JAMIA

JF - Journal of the American Medical Informatics Association : JAMIA

SN - 1067-5027

IS - 6

M1 - ocw005

ER -