Combining augmented reality with deep learning network to help pathologists spot cancer cells

Combining augmented reality with deep learning network to help pathologists spot cancer cells
Left: Schematic overview of the ARM. A digital camera captures the same field of view (FoV) as the user and passes the image to an attached compute unit capable of running real-time inference of a machine learning model. The results are fed back into a custom AR display which is inline with the ocular lens and projects the model output on the same plane as the slide. Right: A picture of our prototype which has been retrofitted into a typical clinical-grade light microscope. Credit: Google

A team of researchers at Google has developed a tool that combines augmented reality with a deep learning neural network to provide pathologists with help in spotting cancerous cells on slides under a microscope. Google has published a paper outlining the new tool, which they describe as an augmented reality microscope (ARM) platform, on their website. They also gave a presentation at this year's American Association for Cancer Research meeting, showing what it can do.

Performing a biopsy entails slicing tiny bits of tissue from a part of the body doctors suspect might be cancerous. That tissue is sliced thinly enough to view on a glass slide under a microscope by a highly trained pathologist looking for . The job of the pathologist, the Google team notes, is highly valued, but the work is also tedious, which can lead to fatigue and errors. To help the pathologist, the team at Google built on prior work at the company in which a team developed a network that was trained to spot breast cancer. The team then incorporated lessons learned from various projects surrounding to train their deep learning application to draw circles around suspected in a tissue sample. They interfaced their system with the type of microscope pathologist already use. The result is an add-on tool for the microscope. It works like this: Through the , the pathologist sees the cells of the tissue—moving the slide around allows them to see the whole sample. But then,if the ARM spots possible cells, it draws a ring around them, alerting the pathologist to take a closer look.

The ARM, Google reports, is now capable of assisting with detecting breast and prostate cancers, but they note that it could be trained to spot other cancers, as well. It could even be trained to look for some types of infectious diseases such as TB and malaria. They note also that it could also be programmed with other AR capabilities, such as posting labels over certain tissue sections, including arrows or other indicators pathologists might find useful.

Explore further

Smart software can diagnose prostate cancer as well as a pathologist

© 2018

Citation: Combining augmented reality with deep learning network to help pathologists spot cancer cells (2018, April 17) retrieved 19 October 2021 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors