In-House LLM Serving at Netflix

This blog post discusses Netflix's in-house LLM serving architecture, which runs the full stack from model deployment to inference within the production environment. It details the technology choices made during the development of the platform, including engine selection (vLLM), model packaging, API design, and deployment strategies. The article emphasizes lessons learned from production that guided decisions in designing a robust and scalable LLM serving system, alongside operational challenges faced and future improvements considered. It reflects a collaborative effort to enhance the machine learning workflow at Netflix.