Wearable cameras allow AI to detect medications

A team of researchers say they have developed the first wearable camera system that uses artificial intelligence to detect potential errors in medication administration.

In a test whose results were published today, the video system recognized and identified with high proficiency which medications were being administered in a busy clinical environment. The AI ​​achieved a sensitivity of 99.6% and a specificity of 98.8% in detecting bottle change errors.

The findings will be reported on October 22 npj Digital medicine.

The system could become a crucial safeguard, especially in operating rooms, intensive care units and emergency medicine, said co-lead author Dr. Kelly Michaelsen, assistant professor of anesthesiology and pain medicine at the University of Washington School of Medicine.

“The thought of being able to help patients in real time or prevent a medication error before it happens is very powerful,” she says. “You can hope for 100% performance, but even humans can’t achieve that. A survey of more than 100 anesthesia providers found that the majority wanted the system to be more than 95% accurate, and that is a goal we have achieved.”

Drug administration errors are the most commonly reported critical incidents in anesthesia and the most common cause of serious medical errors in the intensive care unit. In the bigger picture, an estimated 5% to 10% of all medications given are associated with errors. Adverse events associated with injectable medications affect an estimated 1.2 million patients annually and cost $5.1 billion.

Mistakes when switching syringes and vials most often occur during intravenous injections where a physician must transfer the medication from vial to syringe to the patient. About 20% of errors are substitution errors where the wrong vial is selected or a syringe is incorrectly labeled. Another 20% of errors occur when the drug is labeled correctly but is administered incorrectly.

Safety measures have been put in place, such as a barcode system that quickly reads and confirms the contents of a bottle, to prevent such accidents. But practitioners sometimes forget this check during high-stress situations because it’s an extra step in their workflow.

The researchers’ goal was to build a deep-learning model that, when combined with a GoPro camera, is advanced enough to recognize the contents of cylindrical vials and syringes and appropriately warn before administering the medication the patient enters.

Training the model took months. The researchers collected 4K video of 418 drug administrations by 13 anesthesiology providers in operating rooms where setups and lighting varied. The video features doctors administering vials and syringes of selected medications. These video clips were later recorded and the contents of the syringes and vials were indicated to train the model to recognize the contents and containers.

The video system does not directly read the text on each vial, but scans for other visual cues: size and shape of the vial and syringe, color of the vial cap, print size of the label.

“It was particularly challenging because the person in the OR is holding a syringe and a vial, and you can’t quite see either item. Some letters (on the syringe and vial) are covered by the hands. And the hands move quickly. They do the work. They don’t pose for the camera,” said Shyam Gollakota, co-author of the paper and professor at the UW’s Paul G. Allen School of Computer Science & Engineering.

Furthermore, the computational model had to be trained to focus on medications in the foreground of the frame and to ignore bottles and syringes in the background.

“AI does all that: detecting the specific syringe that the healthcare provider picks up, not detecting a syringe lying on the table,” Gollakota said.

This work shows that AI and deep learning have potential to improve safety and efficiency in a number of healthcare practices. Researchers are just beginning to explore the potential, Michaelsen said.

The study also involved researchers from Carnegie Mellon University and Makerere University in Uganda. The Toyota Research Institute built and tested the system.

The Washington Research Foundation, the Foundation for Anesthesia Education and Research, and a grant from the National Institutes of Health (K08GM153069) funded the work.

The authors have declared their potential conflicts of interest in their article, which will be made available upon request.

Access downloadable video files shows how AI recognizes vial change errors and proper medication transfer in real time.