This blog post discusses a technical challenge faced in Pinterest's next-generation database ingestion framework regarding data completeness and partition finalization. It outlines the significance of knowing when data is finalized in a stream-based system, presenting a new mechanism developed to track partition finalization using event time-based markers. The article delves into the implementation of this mechanism within the Flink-to-Iceberg pipeline, detailing how partition finalization watermarks are derived from processing event time statistics and propagated through the system. Overall, it highlights the improvements brought to the data ingestion process by transitioning to stream-based methods while ensuring data safety for downstream consumers.