👩💻 Join our community of thousands of amazing developers!
The journey of evolving our streaming platform and pipeline to better scale and support new use cases at Lyft.BackgroundIn 2017, Lyft’s Pricing team within our Marketplace organization was using a cronjob-based Directed Acyclic Graph (DAG) to compute dynamic pricing for rides. Each unit in the DAG would run at the top of every minute, fetch the data from the previous unit, compute the result, and store it for the next unit. This paradigm had several minutes of inherent latency.As product iterati...