👩💻 Join our community of thousands of amazing developers!
How we process half a billion web records a day, without spending half a billion everyday. This blog explains a key mechanism in our data platform infrastructure which enables our web data asset to be refreshed hourly and process late arriving web events.Here’s a basic architecture diagram of our data platform:As you can see we use DBT as a workflow and data transformation layer, and BigQuery for data storing and querying. Continue reading......