How did we use DBT and BigQuery to manage late arriving web events?

1 · The Guardian · Oct. 1, 2024, 10:37 a.m.
How we process half a billion web records a day, without spending half a billion everyday. This blog explains a key mechanism in our data platform infrastructure which enables our web data asset to be refreshed hourly and process late arriving web events.Here’s a basic architecture diagram of our data platform:As you can see we use DBT as a workflow and data transformation layer, and BigQuery for data storing and querying. Continue reading......