Significance of Crystallization Experiment Scoring
Scoring is a critical step in any crystallization experiment as it directly impacts the success of determining macromolecular structures. Accurate scoring distinguishes between promising crystals and non-productive conditions, saving valuable time and resources. By systematically evaluating crystal quality and morphology, researchers can prioritize the most promising conditions for further optimization, leading to more efficient structure determination.
Variability in Manual Drop Scoring of Crystallization Experiments
In 2021, we conducted a study to assess the variability in manual drop scoring of crystallization experiments. We provided 1200 images to 7 crystallographers to label manually using the CPOX classification system of MARCO. The study revealed that all seven crystallographers agreed on only about 50% of the images, highlighting significant variability in manual crystallization scoring (Figure 1A).
Figure 1. A) Agreement among crystallographers on general image scoring. B) Agreement among crystallographers on image scoring with crystals
Moreover, we analyzed agreement among crystallographers, specifically for crystal identification. The dataset included 205 images containing crystals, and the results showed unanimous agreement on only 41% of the images (Figure 1B). This indicates that identifying crystals is particularly challenging compared to other image classes. This study underscores the need for an AI-based autoscoring model to handle this time-consuming task, allowing researchers to focus on more critical aspects of their work.
AI-Based Automation as a Solution
Automating the scoring process with Artificial Intelligence (AI) circumvents these stumbling blocks in crystallization experiments. Not only does it offer higher throughput, but it also remains unaffected by factors such as fatigue, haste, or distraction, ensuring consistent performance around the clock. Moreover, when trained on diverse datasets, AI eliminates human bias, resulting in more reliable and accurate image analysis.
Introduction to MARCO
One such AI-based scoring model is Machine Recognition of Crystallization Outcomes, or MARCO, a convolutional neural network-based algorithm developed by Google. Although MARCO integration with Rock Maker® helped reduce time and effort, it has low crystal detection accuracy due to limitations in training datasets and class definitions.
Sherlock by Formulatrix
To address these limitations, Formulatrix has developed an AI-based autoscoring model, Sherlock. Compared to MARCO, Sherlock by Formulatrix is trained on a larger, more diverse practical dataset of over 400,000 images from 41 collaborating laboratories, enabling accurate and reliable crystal identification. Furthermore, it classifies drops into six distinct classes: Crystal, Crystal-Else, Phase Separation, Precipitate, Clear, and Contaminant. The introduction of the Contaminant class enables the model to recognize and filter out images containing artifacts such as dust, plate scratches, or debris that could be mistaken for crystals, effectively cleaning the dataset. Meanwhile, the Crystal-Else class helps correctly identify challenging cases, such as crystals embedded within precipitates, which MARCO often misclassifies. Additionally, Formulatrix’s Sherlock utilizes a morphology-based classification system to identify high-quality crystals suitable for X-ray diffraction analysis (Figure 2).
Figure 2. Image classification system of Sherlock by Formulatrix
Another key factor contributing to Sherlock's improved crystal detection accuracy is the drop detection feature that replaced image tiling. Before an image is processed by the AI model, the drop region is identified by defining a boundary around the individual drop. Sherlock then focuses its analysis on the defined region, minimizing interference from background features outside the drop. This allows the model to evaluate the relevant crystallization area more effectively and reduces background-related noise, improving the reliability of crystal classification (Figure 3).
Figure 3. Drop detection in Formulatrix’s Sherlock
MARCO-Sherlock Performance Comparison
A comparison between MARCO and Formulatrix’s Sherlock, using a diverse dataset of 6573 images from different laboratories, showed that Sherlock outperforms MARCO in overall accuracy (Sherlock 82.7% vs. MARCO 67.1%), crystal recall (90.6% vs. 74.3%), and crystal precision (90.3% vs. 78.3%) (Figure 4).
Figure 4. Performance comparison of Sherlock with MARCO. Overall accuracy represents the agreement between model- and scientist-assigned labels over crystals and non-crystals. Crystal recall indicates the proportion of actual crystal hits that are correctly identified by the model. Crystal precision reflects the proportion of all the images classified as crystals that actually contained crystals
FAQs
What is automated image scoring in protein crystallization experiments?
Automated image scoring in protein crystallization experiments involves using pretrained AI-based autoscoring models to analyze drop images and assign scores based on the likelihood of crystal presence.
How does AI-powered crystallization scoring, such as MARCO and Sherlock, enhance protein crystal detection?
AI-powered crystallization scoring tools like MARCO and Sherlock are trained on extensive and diverse image datasets, enabling them to recognize crystal features with high accuracy. By minimizing human bias and variability, these models enhance the reliability and consistency of protein crystal detection across large datasets.
What are the advantages of AI-based crystallization scoring compared to manual drop evaluation?
Manual drop evaluation is often time-consuming, subjective, and prone to inconsistency, while AI-based crystallization scoring offers faster, unbiased, and highly consistent image analysis. This automation minimizes human error and frees researchers to focus on experimental design and data interpretation rather than repetitive image evaluation.
How reliable is AI-based crystallization scoring compared to manual scoring by expert crystallographers?
The reliability of AI-based crystallization scoring depends on the quality and diversity of the training data used to develop the algorithm. Well-trained models can achieve accuracy comparable to expert crystallographers in scoring crystallization experiments.