Significance of Crystallization Experiment Scoring
Scoring is a critical step in any crystallization experiment as it directly impacts the success of determining macromolecular structures. Accurate scoring distinguishes between promising crystals and non-productive conditions, saving valuable time and resources. By systematically evaluating crystal quality and morphology, researchers can prioritize the most promising conditions for further optimization, leading to more efficient structure determination.
Variability in Manual Drop Scoring of Crystallization Experiments
In 2021, we conducted a study to assess the variability in manual drop scoring of crystallization experiments. We provided 1200 images to 7 crystallographers to label manually using the CPOX classification system of MARCO. The study revealed that all seven crystallographers agreed on only about 50% of the images, highlighting significant variability in manual crystallization scoring.
A) Agreement among crystallographers on general image scoring
B) Agreement among crystallographers on image scoring with crystals
Moreover, we also analyzed the agreement among crystallographers, specifically in crystal identification. The dataset included 205 images containing crystals, and the results showed unanimous agreement on only 41% of the images. This indicates that identifying crystals is particularly challenging compared to other image classes. This study underscores the need for an AI-based autoscoring model to handle this time-consuming task, allowing researchers to focus on more critical aspects of their work.
AI-Based Automation as a Solution
Automating the scoring process with Artificial Intelligence (AI) circumvents these stumbling blocks in crystallization experiments. Not only does it offer higher throughput, but it also remains unaffected by factors such as fatigue, haste, or distraction, ensuring consistent performance around the clock. Moreover, when trained on diverse datasets, AI eliminates human bias, resulting in more reliable and accurate image analysis.
Introduction to MARCO
One such AI-based scoring model is Machine Recognition of Crystallization Outcomes or MARCO, a convoluted neural network-based algorithm developed by Google. Although MARCO integration with Rock Maker helped reduce time and effort, it has low crystal detection accuracy due to limitations in training datasets and class definitions.
Sherlock by Formulatrix
To address these limitations, Formulatrix has developed an AI-based autoscoring model, Sherlock. Compared to MARCO, Sherlock is trained on a larger, more diverse practical dataset of over 800,000 images from 41 collaborating laboratories, enabling accurate and reliable crystal identification. Furthermore, it classifies drops into six distinct classes: Crystal, Crystal-Else, Phase Separation, Precipitate, Clear, and Contaminant. The introduction of the Contaminant class enables the model to recognize and filter out images containing artifacts such as dust, plate scratches, or debris that could be mistaken for crystals, effectively cleaning the dataset. Meanwhile, the Crystal-Else class helps correctly identify challenging cases, such as crystals embedded within precipitates, which MARCO often misclassifies. Additionally, Sherlock utilizes a morphology-based classification system to identify high-quality crystals suitable for X-ray diffraction analysis.
Sherlock Image Classification System
Another key factor contributing to Sherlock's improved crystal detection accuracy is local feature detection or tiling. Before images are sent to Sherlock for scoring, they are divided into smaller sections, or tiles, enabling more detailed analysis to detect small crystals that MARCO might miss.
Image Tiling in Sherlock
MARCO-Sherlock Performance Comparison
A comparison between MARCO and Sherlock, using a diverse dataset of 6662 images from different laboratories, showed that Sherlock outperforms MARCO in overall accuracy (Sherlock 81.27% vs. MARCO 66.96%), crystal recall (71.38% vs. 45.36%), and crystal precision (73.87% vs. 36.8%).
Performance comparison of Sherlock with MARCO. Overall accuracy represents the agreement between model- and scientist-assigned labels over crystals and non-crystals. Crystal recall indicates the percentage of crystals correctly identified as crystals (true positives). Crystal precision reflects the proportion of non-crystals correctly identified as non-crystals
FAQs
What is automated image scoring in protein crystallization experiments?
Automated image scoring in protein crystallization experiments involves using pretrained AI-based autoscoring models to analyze drop images and assign scores based on the likelihood of crystal presence.
How does AI-powered crystallization scoring, such as MARCO and Sherlock, enhance protein crystal detection?
AI-powered crystallization scoring tools like MARCO and Sherlock are trained on extensive and diverse image datasets, enabling them to recognize crystal features with high accuracy. By minimizing human bias and variability, these models enhance the reliability and consistency of protein crystal detection across large datasets.
What are the advantages of AI-based crystallization scoring compared to manual drop evaluation?
Manual drop evaluation is often time-consuming, subjective, and prone to inconsistency, while AI-based crystallization scoring offers faster, unbiased, and highly consistent image analysis. This automation minimizes human error and frees researchers to focus on experimental design and data interpretation rather than repetitive image evaluation.
How reliable is AI-based crystallization scoring compared to manual scoring by expert crystallographers?
The reliability of AI-based crystallization scoring depends on the quality and diversity of the training data used to develop the algorithm. Well-trained models can achieve accuracy comparable to expert crystallographers in scoring crystallization experiments.