«Evaluating Smart Forms and Quality Dashboards in an EHR Inclusive Dates: 09/30/04 - 09/29/09 Principal Investigator: Blackford Middleton, MD, MPH, ...»
Interventions The interventions consisted of two types: Smart Forms and Quality Dashboards. Smart Forms were customized note-writing templates that were built into the LMR. They gathered and organized existing data relevant to the management of patients with ARI or CAD, allowed for physician entry of additional relevant data, provided decision support tailored to current guidelines and patient information, and allowed for documentation, ordering, and updating of LMR data in a single step. They also facilitated tasks such as printing patient instructions and scheduling follow-up appointments.
Quality Dashboards provided a report of all the patients with CAD or ARI seen by a particular physician regarding adherence to recommended guidelines (e.g., aspirin use for patients with CAD, antibiotic over-prescribing in patients with bronchitis). Graphical displays allowed for comparisons among physicians within a practice as well as comparisons to local and national benchmarks. Other features facilitated physician actions such as letter writing and test ordering.
Statistical Analysis All randomized controlled trials data were analyzed on an “intent-to-intervene” basis. That is, any visit occurring in a clinic assigned to Smart Forms or Smart Forms plus Quality Dashboards, whether the physician did or did not use either intervention, was considered an intervention visit in the analysis. We assessed the significance of the Smart Forms or Smart Forms plus Quality Dashboards on the primary outcomes, as appropriate for each condition. A patient in the study clinics could be a subject in the ARI study, the CAD study, both, or neither, depending on whether he/she fulfilled the inclusion criteria for that particular visit. For CAD analyses, the patient was the unit of analysis because we looked into whether the patient met guidelines at the end of the study period, regardless of the number of visits. For ARI analyses, the unit of analysis was the visit because the important question was whether an antibiotic was prescribed at a given visit. Other than its effect on correlation (see below), this difference did not affect the analytic approach.
For dichotomous outcomes (e.g., prescription of aspirin for CAD, antibiotic use in ARI), the outcomes were measured and analyzed on a per patient basis, although conceptually it is simpler to think about the percentage of patients in each group who met the guidelines at the end of the study period. For the three-armed trial, there were two primary comparisons: guideline compliance in each intervention group compared to the control group. Because of the multiple testing issues a Bonferroni adjustment was used to determine significance, so that a p value
0.025 was required for each of the two tests of efficacy. The comparison of the two intervention arms to each other occurred as a secondary outcome.
Univariable analyses of the primary predictor (i.e., intervention group) were performed for the primary outcomes (adherence with each guideline, each of which is a dichotomous variable) using Chi-square or Fisher’s Exact tests in order to get a general sense of the data. Univariable analysis was also performed on potential confounders using standard statistical tests (Chi square or Fisher’s exact test for dichotomous or categorical variables, chi-square test for trend for ordinal variables, Student’s t-test or Wilcoxon rank sum for continuous variables, depending on the normality of the data). Univariable statistical methods were also used to describe the patient population in each treatment arm in terms of demographics and clinical characteristics.
We then built multivariable logistic regression models, with the primary outcome being guideline adherence at the end of the study period. The assigned intervention group was a primary predictor. We built a model adjusting for guideline adherence immediately before the study period and all potential confounders were identified through a significance level of p0.10 from univariable analyses. Potential confounders included patient factors (age, gender, race, median income by zip code, insurance type), physician factors (age, gender, and patient volume), practice factors (size of practice), and predictors unique to each outcome (e.g., number of years since diagnosis of CAD (for all CAD outcomes), signs and symptoms associated with antibiotic prescribing (for ARI outcomes)). Any potential confounders that changed the effect estimates for the intervention covariates by more than 10% were retained as part of the final model.
Multivariable models took the following general form, with the dichotomous variable “intervention” (Smart Form vs. Control; Smart Form plus Quality Dashboard vs. Control) as the
independent variable of interest:
Outcome = intervention + patient variables + physician variables + clinical variables In addition, our CAD analyses took into account a design feature that while the patient is the unit of analysis, the practice site is the unit of randomization and the physician is the main target of the interventions. To adjust for clustering of patients within physicians and physicians within practice sites, we used two-level hierarchical regression models to account for intra-class correlations. For ARI analyses, there are actually three levels of correlation (visits within patients within physicians within practices), so we used three-level hirearchical models. The SUDAAN program was used to carry out the modeling, incorporating an exchangeable correlation structure within patient, physician, and practice site.
For the secondary outcomes, we performed similar univariate and multivariable analyses, using similar statistical techniques as for the primary outcomes. For continuous variables (e.g., LDL cholesterol, blood pressure in CAD), multivariable analyses was performed using linear regression in a similar model-building strategy to that described above for logistic regression.
For cost data (e.g., antibiotic costs in ARI), if the distribution of costs is not normally distributed, we used either a t-test with log transformation of cost or Wilcoxon rank sum in univariable analyses.
To preserve statistical power and reduce the costs of the study, we decided not to include a separate “quality dashboard only” arm. We also view the quality dashboard as a complement to smart form documentation, and not necessarily as a stand-alone intervention. As a rough estimate of quality dashboard’s ability to improve care by itself, we compared each of the main outcomes before and at the end of the pilot period in the practice group that received it.
We used SAS software (SAS Institute, Cary, NC) and SAS-callable SUDAAN statistical software (Research Triangle Institute, Research Triangle Park, NC), which is capable of adjusting for multilevel clustering effects for all analyses.
Evaluation of Outcomes Outcomes were collected electronically from the Partners Central Data Repository. In addition, in a 10% subset of patients in the ARI cohort, medical records were reviewed to validate diagnoses, signs and symptoms, antibiotic use, test results, and comorbid conditions.
ARI Cohort: Primary Outcome. We evaluated antibiotic prescribing rates.
ARI Cohort: Secondary Outcomes. We evaluated appropriateness of antibiotic prescribing;
re-visit rates within 30 days, antibiotic costs, use of broad-spectrum antibiotics, all-cause antibiotic use, rates of different ARI diagnoses, quality of documentation regarding specific ARI signs and symptoms.
CAD Cohort: Primary Outcomes. We evaluated aspirin use (on medication list at end of study period), beta-blocker use, ACE inhibitor use, LDL testing, LDL 100 mg/dL, blood pressure 140/90 mm Hg, Hgb A1c testing and Hgb A1c 7 among diabetics.
CAD Cohort: Secondary Outcomes. We evaluated LDL cholesterol, blood pressure, and Hgb A1c levels as continuous variables; quality of documentation: blood pressure, height, weight, BMI, and smoking status; allergies or intolerance to aspirin, beta-blockers, and ACE inhibitors;
family history of CAD; test results from outside laboratories; reasons for non-adherence with guidelines.
Usability Testing Clinicians sent their comments by email during a 3-month pilot period in which they used the module for the documentation of actual visits. Another set of comments was entered in an online survey at the end of the pilot. We also extracted direct quotes of clinicians from transcripts of interviews and think-aloud study protocols that were completed as parts of usability evaluation.
Although collected through a variety of methods all comments were reviewed and analyzed by usability experts. We analyzed all 155 statements about usability problems collected during the study to identify emergent themes following grounded theory principles. Two researchers then independently assigned the statements into heuristic categories, either general or modified according to newly identified themes. Several iterative coding sessions and discussions ensued, and as a result of extensive comparison and refinement, twelve heuristic categories were formulated.
Ensuring Safety of Participants Study participant safety was assured through our data collection, analysis and monitoring procedures, to preserve patient confidentiality protects their safety. No patients received less than standard of care.
ARI SF Pilot. Ran for 6 weeks and included 16 physicians from within the Partners network (10 Massachusetts General Hospital (MGH) and 6 Brigham and Women’s Hospital (BWH)).
This group of 10 Partners-affiliated physicians represented 9 different practices in the Partners network. Although nurse practitioners were among those invited to participate in the pilot, all 10 participants who used the ARI Smart Form with real patients were physicians. These clinicians included 5 women, had a mean age of 42 (SD±6.7) years old, and, on average, graduated from medical school 15 years previously. Nine of the pilot clinicians had primary care practices and 1 saw only urgent care patients.
The mean age of the 26 patients for whom the ARI Smart Form was used was 44 (SD±15) years old and included 15 (60%) women. Of these patients, 17 (65%) were white, 2 (8%) were Latino, and 7 (27%) had unknown race and ethnicity. Twenty-four patients (92%) spoke English as their primary language.
Overall, during the pilot period, clinicians prescribed antibiotics to 35% (9 of 26) of patients when using the ARI Smart Form and 38% (15 of 39) of patients when not using the ARI Smart Form for ARI visits. For antibiotic-appropriate diagnoses, clinicians prescribed antibiotics in 6 of 6 visits (100%) when using the ARI Smart Form, 9 of 10 visits (90%) when not using the ARI Smart Form compared to 154 of 367 visits (42%) during the previous cold and influenza season.