Debugging the One-in-a-Million Failure: Migrating Pinterest’s Search Infrastructure to Kubernetes

The blog post details Pinterest's complex migration of its search infrastructure, 'Manas', to a Kubernetes platform, highlighting a critical latency issue found during performance testing. The team discovered that cAdvisor's memory-contestation feature caused significant slowdowns, impacting requests at a rate of one in a million. Through meticulous debugging and performance profiling, the team identified the cause and managed to disable the intrusive metric collection from cAdvisor, stabilizing the performance of the search system. This experience provided valuable lessons in debugging and resource management while enabling further migration of services to their new platform.