In a recent study published in Nature Medicineresearchers developed the fundamental model medical concept retriever (MONET), which associates medical images with text and evaluates images based on their concept existence, which helps critical tasks in the development and application of medical artificial intelligence (AI).
Study: Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning. Image credit: LALAKA/Shutterstock.com
Record
Building reliable image-based medical AI systems requires the analysis of information and neural network models at every level of development, from the training phase to the post-development phase.
Richly annotated medical datasets containing semantically relevant insights could demystify “black box” technologies.
Understanding clinically important concepts such as darker pigmentation, atypical pigment networks, and multiple colors is medically beneficial. However, obtaining labels requires effort, and most medical information sets merely provide diagnostic annotations.
About the study
In the current study, the researchers created MONET, an artificial intelligence model that can annotate medical images with medically relevant insights. They designed the model to recognize various human-understandable insights in two image modalities in dermatology: dermoscopic and clinical images.
The researchers collected 105,550 dermatology image-text pairs from PubMed articles and medical textbooks, followed by training on MONET using 105,550 dermatology-related photos and natural language data from a large-scale medical literature database.
MONET assigns scores to photos for each concept, which indicate the degree to which the image illustrates the concept.
MONET, based on adversarial type learning, is an artificial intelligence approach that allows direct application of plain language description to images.
This method avoids manual labeling, allowing bulk information of image-text pairs on a much larger scale than is possible with supervised type learning. After training MONET, the researchers evaluated its effectiveness in annotations and other use cases related to AI transparency.
The researchers tested the annotation capabilities of the MONET concept by selecting the most conceptual photographs from dermoscopic and clinical images.
They compared the performance of MONET with supervised learning strategies that include training ResNet-50 models with conceptual ground-truth labels and OpenAI’s adversarial language image pretraining (CLIP) model.
The researchers also used MONET to automate data evaluation and tested its effectiveness in differential concept analysis.
They used MONET to analyze data from the International Skin Imaging Collaboration (ISIC), the largest dermoscopic image collection of more than 70,000 publicly available images commonly used to train dermatological AI models.
The researchers developed model checking using MONET’ (MA-MONET) using MONET to automatically detect semantically relevant medical concepts and model errors.
The researchers evaluated MONET-MA in real-world settings by training CNN models on data from multiple universities and evaluating their automated concept annotation.
They contrasted the “MONET + CBM” automatic idea scoring method with the human labeling method, which applies only to photos containing SkinCon tags.
The researchers also investigated the effect of concept selection on MONET+CBM performance, especially concepts related to tasks at congestion levels. Furthermore, they evaluated the impact of incorporating the bottleneck red concept on MONET+CBM performance in inter-institutional transfer scenarios.
Results
MONET is a flexible medical AI platform that can appropriately annotate insights into dermatological images as validated by board-certified dermatologists.
The concept annotation feature enables relevant reliability assessments across the medical AI pipeline as evidenced by model checks, data checks, and interpretable model evolutions.
MONET successfully finds suitable dermatoscopic and clinical images for various dermatology keywords, outperforming the base CLIP model in both areas. MONET outperformed CLIP for dermoscopic and clinical images, while remaining equivalent to supervised learning models for clinical images.
MONET’s automated annotation feature helps identify differentiated features between any two arbitrary groups of images in human-readable language during differential idea analysis.
The researchers found that MONET recognizes different expressed ideas in clinical and dermoscopic datasets and can help in large-scale data screening.
Using MA-MONET revealed features associated with high error rates, such as a cluster of photographs labeled blue-white veil, blue, black, gray, and flat-topped.
The researchers identified the cluster with the highest error rate for erythema, regression structure, redness, atrophy, and hyperpigmentation. Dermatologists selected ten target-related concepts for the MONET+CBM and CLIP+CBM congestion layers, allowing flexible labeling options.
MONET+CBM outperforms all mean area under the receiver operating characteristic curve (AUROC) baselines for predicting malignancy and melanoma in clinical images. Supervised black-box models consistently performed better in cancer and melanoma prediction tests.
conclusion
The study found that image-text models can increase the transparency and credibility of artificial intelligence in the medical field. MONET, a platform for medical concept annotation, can improve the transparency and reliability of dermatological AI by enabling large-scale concept annotation.
AI model developers may improve data collection, processing, and optimization processes, resulting in more reliable AI medical models.
MONET can impact the clinical development and monitoring of medical image AI systems by enabling full control and fairness analysis through annotations of skin tone descriptors.
