Load-balanced Brooklin Mirror Maker: Replicating large-scale Kafka clusters at LinkedIn

2 · LinkedIn · April 11, 2022, 9:45 p.m.
At LinkedIn, Apache Kafka is used heavily to store all kinds of data, such as member activity, log storage, metrics storage, and a multitude of inter-service messaging. LinkedIn maintains multiple data centers with multiple Kafka clusters per data center, each of which contains an independent set of data. Mirroring (i.e., replicating) Kafka topics across the clusters and data centers not only enables easy accessibility and analytics by aggregation of data from multiple data centers, but also fau...