Summary
This blog post discusses the challenges of diagnosing discrepancies between offline and online performance metrics of Pinterest's L1 conversion models. The authors, all machine learning engineers at Pinterest, share their structured investigation process, uncovering issues related to feature availability during serving, embedding version skews, and the importance of funnel alignment in the ads system. The conclusion emphasizes a proactive design approach to handling online-offline discrepancies, integrating various model and pipeline considerations to improve predictability of model launches and real-world outcomes.