Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

Summary

The article presents a study on enhancing search relevance in app store rankings by integrating LLM-generated judgments. It identifies the challenge of limited expert-provided textual relevance labels compared to abundant behavioral relevance labels. The authors evaluate various LLM configurations and find that a specialized, fine-tuned model significantly outperforms larger pre-trained models in generating relevant labels. By leveraging this optimized model, they generate millions of textual relevance labels, which, when integrated into the production ranker, lead to improved offline metrics such as NDCG and a statistically significant increase in conversion rates during A/B testing. The findings emphasize the importance of textual relevance in scenarios with limited behavioral data, particularly for tail queries.

Key Learnings

• 1Fine-tuning specialized models can yield better performance in generating relevant labels compared to larger pre-trained models.
• 2Generating textual relevance labels can significantly enhance search ranking systems, especially in data-scarce environments.
• 3The integration of LLM-generated labels into existing ranking systems can lead to measurable improvements in user engagement metrics.
• 4Understanding the balance between behavioral and textual relevance is crucial for optimizing search systems.
• 5A/B testing is essential to validate the effectiveness of new models and approaches in real-world applications.

Who Should Read This

Senior Machine Learning Engineers exploring advanced techniques for search relevance optimization in large-scale systems.

Test Your Knowledge

Topics