How we scaled async workload processing at GitLab.com using Sidekiq

GitLab is a Ruby-on-Rails application that processes a lot of data. Much of this processing can be done asynchronously, and one of the solutions we use to accomplish this is Sidekiq which is a background-processing framework for Ruby. It handles jobs that are better processed asynchronously outside the web request/response cycle. There are a few terms that that we'll use in this post: A worker class is a class defined in our application to process a task in Sidekiq. A job is an instance of a wo...