LLM-Powered Relevance Assessment for Pinterest Search

This blog discusses how Pinterest Search utilizes large language models (LLMs) to enhance the relevance assessment of search results. By fine-tuning LLMs on human-annotated data, they can efficiently evaluate search ranking through A/B testing, significantly reducing labeling costs and improving the overall user experience. The methodology includes a stratified sampling design to measure heterogeneous effects in search relevance and shows strong alignment with human labels, highlighting the effectiveness of LLMs across multiple languages.