[RFC] Support for horizontal scaling Sourcebot

Start by mapping the webapp, worker, and zoekt services and reviewing the proposed docker-container.yml and docker-compose.yml deployment entry points. Investigate what each service requires to run multiple replicas, then compare local Compose with the proposed Helm, Terraform, and NixOS production paths. Done requires an agreed design, sensible defaults, and documented deployment configurations.

This discussion tracks adding support for horizontal scaling to Sourcebot.

TL;DR

• I think the existing single, self-contained docker container is great since it's super easy to use and is radically simple.
• But it's probably not practical in larger deployments, especially ones with thousands of repositories.
• We've seen instances where when a machine is under heavy indexing load, the webapp can become unresponsive or even OOM errors can occur.
• I think the first logical step here is to distribute our services as separate images (i.e., ghcr.io/sourcebot-dev/(webapp|worker|zoekt))
• Upstream dependencies like Postgres, and Redis would be separate.
• More images = more complexity, so I think the best mechanism to manage this complexity is moving towards declarative configs. We could do the following:
  • For local deployments, we can define a docker-container.yml at the root of the repository. Deploying will now involve 1) Cloning the repo, and 2) running docker compose up.
  • For production deployments, we can use technologies such as Helm, Terraform, Railway, and NixOS
• I think it will be important to have sensible defaults (s.t., it's very easy to deploy), while having deep configuration options that are well documented (s.t., it fits the user's deployment story).

Open questions / investigations

• What will we need to do to make each one of our services (webapp, worker, zoekt) to support multiple replicas at the same time?

Is this Overkill?

I'm trying not to go too overboard with the design here. DX should not be sacrificed with this change: ideally it should improve. Sourcebot should be just as easy to deploy as a vertically scaling service, both locally and in production. Deploying it as a horizontally scaling service in production should also be easy, while including configuration depth to support a wide range of deployment scenarios.

Locally, using Docker Compose will streamline deployment since a) it's a single command, docker compose up, and b) it's declarative, so all of the options (w/ sensible defaults, or commented out) can be included inside the docker-compose.yml.

In production, using Helm, Terraform, NixOS, etc. will help get people setup quickly. Again, sensible defaults here are key.

We also should continue supporting the self-contained image since to avoid introducing a breaking change here. We can evaluate if it's still worth shipping whenever we do a v5.