«Evaluating Smart Forms and Quality Dashboards in an EHR Inclusive Dates: 09/30/04 - 09/29/09 Principal Investigator: Blackford Middleton, MD, MPH, ...»
Ten pilot clinicians responded to the post-pilot survey. Three clinicians felt the ARI Smart Form was marginally timesaving, 5 felt it was time-neutral, 1 felt it marginally increased work, and 1 felt it significantly increased work. Six out of the ten clinicians would recommend that other clinicians use the ARI Smart Form unmodified and 3 would recommend it with some minor modification, such as increasing flexibility with more “freelance choices” and making the final note “flow more naturally.” ARI SF RCT. Ran from November 3, 2005 to May 31, 2006, during which period patients made 21,961 ARI visits to study clinics. This study yielded results on antibiotic prescribing rates as well as ARI guidelines familiarity. In the intent-to-intervene analysis, clinicians prescribed antibiotics to 43% of patients with ARI diagnoses in control clinics and to 39% of patients with ARI diagnoses in intervention clinics (odds ratio [OR], 0.8; 95% confidence interval [CI], 0.6 to 1.2; p =.30). There was no significant difference in antibiotic prescribing for antibiotic-appropriate ARIs (OR, 0.8; 95% CI, 0.5 to 1.3) or for non-antibiotic appropriate ARIs (OR, 0.9; 95% CI, 0.6 to 1.4).
ARI SF Usage. In 27 PHS affiliated intervention clinics, 33% (86/262) of clinicians used the ARI Smart Form at least once. Based on ICD-9 codes, the ARI Smart Form was used in 6% (742/11,954) of ARI Visits. For intervention ARI visits at which the ARI Smart Form was used, the duration of ARI Smart Form use was 8.1 (standard deviation, 5.8) minutes.
ARI QD RCT was completed August 31, 2007. Data has been retrieved from the Partners Central Data Repository and data analysis is nearly complete.
CAD/DM SF Pilot. Ran from 3/7/2006 to 5/16/2006 and involved 30 clinicians. For this pilot all SF users went through SF in-person training sessions (from 3/07/2006 to 3/31/06). For those users who could not attend, we made an online “Robodemo” instruction tutorial available.
The majority of the participants were primary care physicians (77%) but also included specialists (4 endocrinologists and one cardiologist), and nurse practitioners (2). Most participants were attending physicians, with a mean of 20 years since graduation from medical school. Clinicians saw an average of 27 patients per week; this number reflects the high number of clinicians in the Partners system who practice part-time. As expected, self-reported experience with using the LMR varied, with 23% very experienced and 20% somewhat or very inexperienced.
Approximately one quarter of clinicians usually or always wrote their visit notes in the LMR during the patient visit.
During the pre-intervention and intervention periods, clinicians saw 1940 patients with CAD and/or DM. Patients were 51% women and had a mean age of 65 years; 79% of the patients had DM, 35% had CAD, and 14% had both. Compared with patients during the intervention period for whom the Smart Form was not used, those patients for whom the Smart Form was used were more likely to have a managed care insurance plan, less likely to have Medicare, less likely to have CAD, more likely to have DM, and had fewer medical problems documented in the LMR Problem List.
During the intervention period, 21 participants (70%) used the CAD/DM Smart Form at least once. Seventeen participants (57%) used the Smart Form with two or more patients, and two participants opened the Smart Form during ten or more patient visits during the 6-8-week period.
During the intervention period, the Smart Form was used with 150 patients, while there were 935 visits by patients with CAD or DM during the intervention period in which the Smart Form was not used (i.e., the Smart Form was used in approximately 14% of eligible visits).
Sixty-one percent of participants completed the pre-pilot survey and 48% completed the postpilot survey. Survey responses suggested a trend towards improved participants’ satisfaction with their management of smoking, ACE I/ARB use, and especially diet and exercise, but these differences were not statistically significant. In the post-pilot usability survey, the majority of participants agreed that the CAD/DM Smart Form helped them to improve compliance with clinical guidelines and improve the quality of patient care. Survey results also showed that pilot participants do not currently consider the CAD/DM Smart Form to be a timesaver or a tool to
improve their workflow. Users reported the following Smart Form features as most helpful:
organizing data, providing assessments for each area of disease management, providing suggested orders based on individual patient data, and printing patient instructions.
Pre-RCT Online Survey. For both ARI and CAD/DM Smart Form, survey was distributed to 976 clinicians in order to address the secondary goal of evaluating clinical guideline adhirence.
This survey was used to ascertain clinicians’ demographics, workflow, typical use of the Longitudinal Medical Record (LMR, an electronic health record used at Partners institutions), and opinions regarding decision support tools available in the LMR. The survey was with 257 clinicians responding (response rate - 45%).
CAD/DM SF RCT. Ran for 310 days (March 3, 2007 to May 10, 2008; it encompassed 10 PHS Primary Care Practices with 239 physicians and 7009 patients. Data was collected from over 26,000 visits. All data has been retrieved from the Partners Central Data Repository and preliminary analysis completed. Although more extensive analysis is being conducted presently, preliminary results indicate that patients of CAD/DM Smart Form users were more likely to have deficiencies in care addressed. The analysis showed up to date patient BP result in 32% of patients in the intervention group compared to 24% in the control group (p0.001), up to date height/weight result in 5% of patients in the intervention group compared to 4% in the control group (p0.001). Overall percentage of deficiencies addressed in the intervention group was 11.4% compared to 10.1% in the control group (p0.001). Even when the Smart Form was not used, performance in the non-intervention group (using existing CDS tool, EOV) was slightly better than in control group. An up to date patient BP result was shown in 30% of nonintervention group patients compared to 24% of control group patients (p0.001). This result may indicate an influence of Smart Form on PCP awareness and awaits further analysis. Results are summarized in Tables 3-5.
CAD/DM Post-RCT Survey. Showed a response rate of 36% (N=57). Seventy percent of the respondents (N=40) to the survey used the SF on a regular basis, and were thus able to provide answers to all questions on the survey. Of the regular users, 82% agreed with the recommendations the SF provided, 64% agreed that SF helped them comply better with CAD/DM guidelines, while 47% believed SF changed what they normally would have done for their patients’ blood pressure, cholesterol, or glycemic control. In addition, more than half of the
respondents found features in the CAD/DM SF that facilitated patient education to be useful:.
66% of respondents found the patient instruction handout feature to be helpful and 56% found the “Patient View” feature helpful (See CAD/DM Smart Form screenshots, Figure 2a-b).
CAD QD RCT. Ran from March 24, 2008 to March 31, 2009 and encompassed 15 ambulatory primary care practices. A post-RCT survey was distributed to 76 providers.
Relevant results are presented in Table 6 and Figure 1.
SF Usability Testing. Was performed concurrently with CAD/DM pilot testing. Two scenarios for standardized patients were used in the usability testing to compare physicians typical way of documentating a visit; one documenting with the SF during a visit and one documentintg after a visit. There were 155 comments from 36 clinicians obtained either in the form of written communication (email and survey) or transcribed from direct verbal quotes (interview and evaluation). We received 85 emails from nine clinicians (reflecting a 50% response rate), and 20 free-text comments were entered in the online survey by 15 clinicians (54% response). Six clinicians who participated in usability evaluations made 26 comments and another six clinicians made 24 distinct comments during interviews. Over a half of all responses (55%) were emails, and about equal numbers were obtained from the survey, evaluations and interviews (15%, 13% and 17%, respectively). The most common form of a response that constituted about a third of collected data (N=54) was an email classified as either a Biomedical, Control or Fault category.
Comments from other survey sources were most likely to be classified in the following categories: Customization and Control for survey (N=9, 45%), Transparency and Workflow for evaluations (N=14, 54%), and Cognition and Workflow for interviews (N=13, 54%). Overall, the Control, Cognition and Biomedical categories described about a half of all data (52%), and about a third (35%) was classified in the Customization, Workflow and Technical categories.
There were no Consistency or Context comments.
There were 47 findings extracted from expert reports. Over two thirds were classified into just three categories: Cognition, Customization and Workflow. In contrast, none were in the Fault, Speed or Terminology categories and only one was classified as Biomedical. Technical and biomedical concepts were generally not represented in the evaluations. We contrasted all 47 findings with a subset of 105 comments that included only email and survey. Findings were derived from reports of evaluation and interviews that already contained reinterpreted verbal comments of the subjects. We therefore excluded comments made during evaluations from the comparison.
The Smart Form represents documentation-based clinical decision support that goes beyond standard interruptive methods by dynamically rendering an integrated data review, clinical documentation, and decision support environment for the end-user. Critical to the success of this application’s development (and critical lessons for EMR developers and vendors) were strong participatory design principles, iterative development, and an understanding of clinician workflow and psychology. By integrating decision support into a clinician’s workflow, the Smart Form has the potential to facilitate documentation of coded, actionable data, improve the quality of decision-making, and improve the management of patients with acute and chronic medical conditions.
Continued Work In the course of this project we have developed two novel tools for integrated documentationbased decision support – Smart Form and Quality Dashboard. We have implemented these systems at ambulatory primary care settings and evaluated their usage and impact on clinicians’ workflow. For the purpose of this study we focused on three clinical areas – ARI, CAD and DM.
As described above, four RCT studies were conducted in the duration of this five-year project. Complete data sets were retrieved and are presently being analyzed by the project team.
There is a certain trend emerging from the study results up to this point even though the data analysis is still ongoing. Overall, use of Smart Forms and Quality Dashboards as a part of clinical decision support correlates with better adherence to the clinical guidelines within the clinical areas described. Also, most users found these tools intuitive to use, easily integrated into clinicians’ workflow and beneficial in terms of quality of patient care.
Looking forward, these are our plans for the next 6-12 months:
1. Completion of data analysis of ARI quality dashboard as a reporting tool
2. Completion of data analysis CAD quality dashboard as a reporting tool
3. Evaluation usage and usability of the QD application
4. Identification of potential barriers to use of the quality dashboard in clinical decision support
5. 9 manuscripts are being worked on by the leading investigators of our group to be submitted for publication in peer-reviewed journals.