Summative usability testing is an industry standard usability methodology (see http://bit.ly/13WYqTU ) During the testing a trained user experience professional works with the EHR vendor to create a series of representative tasks and then measures the amount of time it takes to complete the task (time on task), how many and what type of errors occur (error rates), and satisfaction with the task interaction (user satisfaction). The results —as measured by things like task times, assisted and un-assisted completion rates and standardized satisfaction scores --can be benchmarked against current industry standards.
Summative testing for ONC certification needs to test the following 9 areas:
- Section 170.315(a)(1) Computerized provider order entry – medications
- Section 170.315(a)(2) Computerized provider order entry – laboratory
- Section 170.315(a)(3) Computerized provider order entry – diagnostic imaging
- Section 170.315(a)(4) Drug-drug, drug-allergy interaction checks
- Section 170.315(a)(5) Demographics
- Section 170.315(a)(9) Clinical decision support (remove 1/1/25)
- Section 170.315(a)(14) Implantable device list
- Section 170.315(b)(2) Clinical information reconciliation and incorporation
- Section 170.315(b)(3) Electronic prescribing
- Section 170.315(b)(11) Decision support interventions
In accordance with NISTIR 7804, ONC requires that the test scenarios be based upon an analysis of critical use risks for patient safety, which can be mitigated or eliminated by improvements to the user interface design.
How should the Results Presented?
Results from the summative study should be created and presented to the ONC using the Customized Common Industry Format (CCIF) Template for EHR Usability Testing (NISTIR 7742 see http://1.usa.gov/gAZFDt ) These reports typically include an executive summary, an introduction, a method section, discussion of the results and any appendices .
ONC now requires that Health IT developers include the specific information from the NISTIR 7742 “Customized Common Industry Format Template for Electronic Health Record Usability Testing. All of the data elements described in NISTIR 7742 and every section must be completed, including “major findings” and “areas for improvement.”
ONC identified the requirements for the submission of the demographic characteristics of test participants. These test participant characteristics must reflect the audience of current and future users of the EHR.
The following are required for submission to the ONC:
- Name and version of the product
- Date and location of the test
- Test environment
- Description of the intended users
- Total number of participants
- Description of participants as follows:
- Sex
- Age
- Education
- Occupation/role
- Professional experience
- Computer experience
- Product experience
- Description of the user tasks that were tested and association of each task to corresponding certification criteria
- List of the specific metrics captured during the testing
- Task Success (%)
- Task Failures (%)
- Task Standard Deviations (%)
- Task Performance Time
- User Satisfaction Rating (Scale with 1 as very difficult and 5 as very easy)
- Test results for each task using metrics listed above
- Results and data analysis narrative:
- Major test finding
- Effectiveness
- Efficiency
- Satisfaction
- Areas for improvement
The Usability People conduct and report summative usability evaluations using the NISTIR 7742 Customized Common Industry Format Template for EHR Usability Testing—suitable for satisfying the Safety-enhanced Design criteria portion of your 2015 Cures Update Edition certification.