Automatic quality improvement reports in the intensive care unit: One step closer toward meaningful use

Abstract

AIM: To examine the feasibility and validity of electronic generation of quality metrics in the intensive care unit (ICU).

METHODS: This minimal risk observational study was performed at an academic tertiary hospital. The Critical Care Independent Multidisciplinary Program at Mayo Clinic identified and defined 11 key quality metrics. These metrics were automatically calculated using ICU DataMart, a near-real time copy of all ICU electronic medical record (EMR) data. The automatic report was compared with data from a comprehensive EMR review by a trained investigator. Data was collected for 93 randomly selected patients admitted to the ICU during April 2012 (10% of admitted adult population). This study was approved by the Mayo Clinic Institution Review Board.

RESULTS: All types of variables needed for metric calculations were found to be available for manual and electronic abstraction, except information for availability of free beds for patient-specific time-frames. There was 100% agreement between electronic and manual data abstraction for ICU admission source, admission service, and discharge disposition. The agreement between electronic and manual data abstraction of the time of ICU admission and discharge were 99% and 89%. The time of hospital admission and discharge were similar for both the electronically and manually abstracted datasets. The specificity of the electronically-generated report was 93% and 94% for invasive and non-invasive ventilation use in the ICU. One false-positive result for each type of ventilation was present. The specificity for ICU and in-hospital mortality was 100%. Sensitivity was 100% for all metrics.

CONCLUSION: Our study demonstrates excellent accuracy of electronically-generated key ICU quality metrics. This validates the feasibility of automatic metric generation.

Key Words: Electronic medical record; Quality indicators; Critical care; Information processing; Datamart; Automatic; Intensive care; Health care

Core tip: Meaningful use of electronic healthcare records (EHRs) requires quality measures. Many administrative reporting tools provided by current EHRs are based on insufficiently accurate data and thus of limited use. We examine the feasibility and the validity of electronic generation of institutional key intensive care unit (ICU) quality metrics using ICU DataMart, a near-real time relational database.

INTRODUCTION

Up to 6 million adult patients are admitted annually to intensive care units (ICU) across the United States[1,2]. This environment is influenced by factors such as intense cognitive demands, multiple communication channels, interruptions, multitasking, and complex organ support devices[3,4]. The distinctive workflow and time sensitive nature of critical illness management are predisposing factors for medical errors[5,6].

According to an Institute of Medicine (IOM) report, health information technology (HIT) must play a central role in the quality improvement process of healthcare[7,8]. Quality measures are required for meaningful use of electronic health records (EHRs)[9]. ICU quality metrics focus on care processes such as adherence to best practices[10,11]. In comparison to these care processes, there are structural measures incorporated in the National Quality Forum (NQF) as quality measures. Metrics can be a valuable tool to evaluate the quality of care delivered to the patient and serve as an indicator to assess whether a patient received recommended steps of care[11,12]. The reporting of valuable quality measures drives clinical process improvements and is largely accepted by clinicians[13].

HIT offers the opportunity to conduct large scale quality improvement projects that were previously limited in scope due to the reliance on manual data abstraction. Manual generation of quality improvement metrics causes dissatisfaction and consumes time and human resources[14,15]. Current clinical information systems do not routinely support the generation of practice management reports. However, the federal Electronic Health Record Incentive Program expects embedded electronic reporting of Quality Metrics directly from EHRs[16]. Automated quality reporting saves 5 to 14 min per case for different metrics[17]. This translates to a theoretical savings of $1 million annually, assuming 100% accuracy of data extraction.

Most high level administrative reporting is based on flawed data and thus of limited clinical use[18,19]. Most of the elements known to be important indicators of processes of care are either not captured, or are captured with insufficient accuracy[18]. The EMR, on the other hand, is a rich source of pertinent information. We recently developed and successfully tested fully automatic APACHE and SWIFT score calculator[20,21]. These study examines the feasibility and the validity of electronic generation of key ICU quality metrics.

MATERIALS AND METHODS

The Critical Care Independent Multidisciplinary Program (CC-IMP) team at Mayo Clinic defined 11 key ICU metrics for electronic reporting. Multiple variables required for electronic data abstractions were identified (Table 1).

Table 1 Critical Care Independent Multidisciplinary Program defined metrics.

Metric		Description	Variables needed
M1	Hospital length of stay for ICU graduates - unadjusted	Hospital length of stay is based on all patients discharged from the hospital during the specified time-frame. A patient may have multiple admissions to the ICU	Hospital Admission Hospital Discharge ICU admission ICU discharge
M2	ICU length of stay - unadjusted	ICU length of stay is based on all patients discharged from the ICU during the specified time-frame	ICU admission ICU discharge
M3	ICU length of stay - adjusted	Adjusted ICU length of stay is based on all patients discharged from the ICU during the specified time-frame. Expected ICU length of stay is the APACHE IV predicted ICU length of stay. Adjusted length of stay is observed ICU length of stay divided by expected ICU length of stay	ICU admission ICU discharge APACHE IV bundle (demographic, vitals, labs, Glasgow coma score, health conditions, time stamps and geolocations, procedures and diagnosis)
M4	ICU readmission rate	Readmission summary is based on all patients admitted to the ICU during the specified time-frame. An admission is counted as a readmission if it is not the patient’s first admission to the ICU during that hospital stay. Readmissions to the same ICU and readmissions within 24 h are summarized	ICU admission ICU discharge Admission source (location)
M5	ICU admissions	Admission summary is based on all patients admitted to the ICU during the specified time-frame	ICU admission ICU discharge
M6	ICU admission source and admission service	Admission source summary is based on all patients admitted to the ICU during the specified time-frame	ICU admission ICU discharge Admission source (location)
M7	Duration of mechanical ventilation	Ventilation summary is based on all patients discharged from the ICU during the specified time- frame. Use of invasive and non-invasive ventilation is summarized as well as the median duration. Numbers for invasive and non-invasive ventilation will not sum to the numbers for ventilation because patients may have both types	ICU admission ICU discharge Duration invasive or non-invasive ventilation for each day of ICU stay
M8	ICU mortality rate - unadjusted	Mortality rate summary is based on all patients discharged from the ICU during the specified time frame. Multiple discharges per patient are allowed	ICU admission ICU discharge Patient status
M9	ICU mortality rate - adjusted	Mortality summary is based on all patients discharged from the ICU during the specified time-frame. Multiple discharges per patient are allowed. Expected ICU mortality is based on each patient’s APACHE IV predicted probability of hospital death. Standard mortality ratio is calculated by dividing observed ICU mortality by expected APACHE IV hospital mortality	ICU admission ICU discharge Patient status APACHE III/IV bundle (demographic, vitals, labs, Glasgow coma score, health conditions, time stamps and geolocations, procedures and diagnosis)
M10	ICU admissions for low-risk monitoring	Low-risk monitoring summary is based on all patients admitted to the ICU during the specified time-frame. Low risk monitoring calculation based on the TISS-28 score[27]. Patients with score 0-13 are considered low-risk monitoring	ICU admission ICU discharge TISS-28 score bundle (vitals, labs, orders, procedures)
M11	ICU census - hourly utilization	Unit utilization summarizes the average hourly unit utilization for each specified time-frame on the hourly ending census	ICU admission ICU discharge Number of bed per unit

ICU: Intensive care unit.

Patient population

Adult patients (age ≥ 18 years) admitted to the medical or surgical ICUs from 04/01/2012 through 04/30/2012 were included. Patients without research authorization were excluded. Out of 925 eligible patients, 93 (10%) were randomly selected for validation purposes. The Mayo Clinic Institution Review Board approved this minimal risk study.

Manual data extraction

All data necessary to calculate quality metrics was collected manually from the EMR by a trained researcher. Standard operating procedures were created to insure uniformity of data collection. Whenever a value was missing or unavailable it was assumed to be normal. The investigator involved in manual data extraction was blinded to the results of the electronic data extraction. Manually collected data was considered as the reference standard.

Electronic data extraction

Information required for quality reports was abstracted from Multidisciplinary Epidemiology and Translational Research in Intensive Care (METRIC) ICU DataMart, a near-real time relational database with open-schema data feeds imported from the EHR[22]. Data domains included physiologic monitoring, laboratory and radiologic investigations, medication orders, and provider notes. An automatic algorithm was used to identify the components required for each of the 11 metrics in the quality reports.

Statistical analysis

Descriptive data are presented as median with interquartile range (IQR) for continuous variables or count with percentage for categorical variables. Paired t-test and McNemar’s tests were used to compare continuous and categorical variable between electronic and manual data. Sensitivity and specificity for electronic data abstraction were calculated based on comparison of the test results and the reference standard (manually created report). All statistical analysis was performed in JMP (SAS, Cary, NC, United States). These statistical methods were reviewed by a statistician from the Division of Biomedical Statistics & Informatics at Mayo Clinic in Rochester, MN.

RESULTS

All types of variables needed for metric calculations were found to be available for manual and electronic abstraction, except information for availability of free beds for patient-specific time-frame (Table 2).

Table 2 Clinical characteristics of baseline cohort.

Number	Characteristic1	Overall (n = 93)
1	Age (yr)	73 (62, 83)
2	Male sex	53 (57%)
3	APACHE III score	60 (46.5, 72)
4	SOFA score	3 (2, 5)
5	ICU type
	Medical	51 (56%)
	Surgical	29 (30%)
	Mixed	13 (14%)
	Ventilator use
6	Invasive	18 (19.4%)
7	Non-invasive	22 (23.7%)
8	ICU length of stay (d)	1.1 (0.9, 2.1)
9	ICU mortality	1 (1%)
10	Hospital length of stay (d)	5.4 (3.6, 11.1)
11	Hospital mortality	10 (10.7%)

¹Values are n (%) for categorical variables and median (IQR) for continuous variables. ICU: Intensive care unit.

There was excellent agreement for admission, discharge, and transfer information, with the exception of ICU discharge time, which differed by more than 1 h in 10.7% of admissions. ICU and hospital mortality were reported identically in both cohorts, with absolute sensitivity and specificity (Table 3).

Table 3 Agreement of admission, discharge, and transfer information.

Component	Agreement
Admit/discharge times1
ICU admission	99%
ICU discharge	89%
Hospital admission	100%
Hospital discharge	100%
Pre-ICU location	100%
Admission ICU	100%
ICU discharge location	100%

¹Agreement was considered to be within ± 1 h. ICU: Intensive care unit.

No statistically significant difference was found between manual and electronic cohorts comparing the use of invasive and non-invasive mechanical ventilation. One hundred percent of sensitivity was found for both variables. One false-positive result was returned from each type of ventilation usage, which produced decreased specificity (Table 4).

Table 4 Analysis of sensitivity and specificity.

Variables and their usage for metrics	Sensitivity	Specificity	True positive	False positive	True negative	False negative
MV use (from M7)	100%	94.40%	75	1	17	0
NIV use (from M7)	100%	95.50%	71	1	21	0
ICU mortality (M8, M9)	100%	100%	0	0	93	0
In-hospital mortality	100%	100%	10	0	83	0

M7, M8, M9 are quality metrics 7, 8 and 9 respectively. ICU: Intensive care unit.

DISCUSSION

The Mayo Clinic EHR contains all necessary data for automatic abstraction and calculation of key quality metrics for ICU practice except availability of free beds and APACHE III admission diagnosis. Demographic information, time stamps for admission and discharge, point-of-care related events, vital signs, laboratory findings, procedures, and outcomes were successfully extracted.

If integrated systems like METRIC ICU DataMart are not used, it is challenging to extract data for automated calculations of quality metrics. Dykes et al[22] reported 41% of variables available from “draft documentation” and an additional 30% available from specific applications, resulting in only 71% of data needed for automated calculation of their custom quality metrics. Kaiser Permanente reports partial or full automation of 6 of 13 quality metrics defined by the Joint Commission with 35% to 61% of data availability[17]. More specific automated quality metric calculation has been reported for SCIP-VTE-2 (deep venous thrombosis) with an accuracy of 96.3%[23]. Measurement concepts have been proposed for automated calculation of American College of Emergency Physicians Quality Metrics basing on availability of variables[24]. EHR-based automated calculation of 12 “meaningful use” quality metrics for more than 100 patients resulted in variable sensitivity (from 46% to 98%) and specificity (62% to 97%). Positive and negative predictive value ranged from 32% to 99%[25].

We used a different approach to provide metrics. Data, organized specifically for analytical purposes in ICU DataMart, provided accurate information about time-specific events during hospitalization[3,4,26]. Using data independent from the clinical practice EHR infrastructure is more flexible. This allows for creation of specific rules and datasets generating algorithms for automated calculation of custom or externally reportable quality metrics. Once data are extracted, they are organized in specific tables in ICU DataMart.

Attempts to analyze the agreement between manual and automated data revealed some sources of data discrepancy. Disagreement for the time of ICU discharge was 11%. This discrepancy resulted from a technical issue: Discharge time for automated data collection was initially defined when monitoring was discontinued, while in manual data abstraction, discharge was defined as the time discharge orders were written. These metrics have been useful for planning purposes as well.

Specific real time metrics were obtained and stored in ICU DataMart tables for retrospective analysis. For example, the current practice relies on this data for bed allocation as service line practices change over time. As a practice grows and requires more beds, our hourly bed utilization guides these changes.

These results demonstrate the feasibility of automated calculation of quality metrics with good accuracy, specificity, and sensitivity. ICU DataMart-based automated calculations can be utilized as a stand-alone tool for practice improvement, administrative purposes, or to provide externally reportable metrics if the institutional EHR does not support such activities.

This retrospective study has several limitations. It was performed in a single center. A custom, institution-specific ICU DataMart for data abstraction and specially developed quality metrics were used.

Automated Quality Metric calculations are feasible using a near-real time EHR infrastructure with data storage capabilities. Electronic data extraction provided accurate (≥ 90%), sensitive (100%), and specific (≥ 93%) results. No user interaction was required to obtain ICU quality metrics.

ACKNOWLEDGMENTS

Authors are grateful to Andrew C Hanson for his assistance and suggestions on preparing the manuscript for publication.

COMMENTS

Background

In the United States millions of adult patients are admitted to intensive care units annually. Health information technology plays a central role in the quality improvement process of healthcare. Intensive care unit (ICU) quality metrics focus on care processes such as adherence to best practices. These metrics can be a valuable tool to evaluate the quality of care delivered to the patient. The reporting of valuable quality measures drives clinical process improvements. Manual generation of quality improvement metrics causes dissatisfaction and consumes time and human resources. The current generation of clinical information systems does not routinely support the generation of practice management reports.

Research frontiers

Most of the high level administrative reporting is of limited use due to outdated, corrupted, and/or incomplete data. There is no universal approach to build automated quality metric reports with direct abstraction from electronic medical records.

Innovations and breakthroughs

A near-real time electronic healthcare record (EHR) infrastructure with data storage capabilities was used for automated quality metric calculations without user interaction. Electronic data extraction provided accurate, sensitive, and specific results.

Applications

This approach can be used for automated quality metric calculation at any institution, regardless of the current EHR system.

Peer-review

Meaningful use of electronic healthcare systems allows immediate data information for both quality-performance and risk management successful processes. Quality metric measures incorporated by official regulatory offices as National Quality Forum enable a high quality course to improve effective choices for an adequate administration in ICU assistance. This study of Mayo Clinic quality-performance protocols supports an optimal use of both human and technical resources, leading us to improved management results as in top-ranked health institutions.