Merge branch 'master' into Update-Caddy-Image

Author: DaedalusG

README.md

@@ -1,87 +1,36 @@
-# Pure-Docker Sourcegraph cluster deployment reference
+# Sourcegraph with Docker Compose
 
-[![sourcegraph: search](https://img.shields.io/badge/sourcegraph-search-brightgreen.svg)](https://sourcegraph.com/github.com/sourcegraph/deploy-sourcegraph-docker)
+[![sourcegraph: search](https://img.shields.io/badge/sourcegraph-search-brightgreen.svg)](https://sourcegraph.com/github.com/sourcegraph/deploy-sourcegraph-docker) [![Build status](https://badge.buildkite.com/e60f9ffcafd68882d3db6fe5e33567e3a111d391a554d50d82.svg)](https://buildkite.com/sourcegraph/deploy-sourcegraph-docker)
 
-There are two recommended ways to deploy Sourcegraph:
+This repository is the deployment reference for [deploying Sourcegraph with Docker Compose](https://docs.sourcegraph.com/admin/install/docker-compose).
 
-- [Single-machine Docker deployment](https://docs.sourcegraph.com/admin/install/docker)
-- [Multi-machine Kubernetes deployment](https://docs.sourcegraph.com/admin/install/kubernetes_cluster)
+> 🚨 IMPORTANT: When upgrading Sourcegraph, please check [upgrading docs](https://docs.sourcegraph.com/admin/updates/docker_compose) to check if any manual migrations are necessary.
+>
+> The `master` branch tracks development. Use the branch of this repository corresponding to the
+> version of Sourcegraph you wish to deploy, e.g. `git checkout 3.19`.
 
-But what if your organization wants a multi-machine deployment without using Kubernetes? What if you use a different container management platform, for example? This project aims to solve that, by providing a pure-Docker deployment option.
+For product and [pricing](https://about.sourcegraph.com/pricing/) information, visit
+[about.sourcegraph.com](https://about.sourcegraph.com) or [contact
+us](https://about.sourcegraph.com/contact/sales) for more information. If you're just starting out,
+we recommend running Sourcegraph as a [single Docker
+container](https://docs.sourcegraph.com/#quickstart-guide) or using [Docker
+Compose](https://docs.sourcegraph.com/admin/install/docker-compose). Migrating to Sourcegraph on
+Kubernetes is easy later.
 
-The goal is that anyone using a container management platform other than Kubernetes (Netflix's [Titus](https://netflix.github.io/titus/), Apache's [Mesos](http://mesos.apache.org/documentation/latest/docker-containerizer/), etc.) would be able to use this repository as a reference for how to deploy Sourcegraph.
+## Is Docker Compose the right deployment type for me?
 
-## Deploying
+Please see [our docs](https://docs.sourcegraph.com/admin/install) for comparisons of deployment types and our resource estimator.
 
-First clone the repository, then:
+## Contributing
 
-```bash
-./deploy.sh
-```
+We've made our deployment configurations open source to better serve our customers' needs. If there is anything we can do to make deploying Sourcegraph easier just [open an issue (in sourcegraph/sourcegraph)](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=deploy-sourcegraph-docker&title=%5Bdeploy-sourcegraph-docker%5D) or a pull request and we will respond promptly!
 
-Visit http://localhost:3080 to visit the running Sourcegraph instance!
-
-## Tearing down the deployment
-
-```bash
-./teardown.sh
-```
-
-## System topology
-
-To understand the system topology:
-
-1. Look at `deploy.sh` to get an overview of services.
-2. Every service (`deploy-*.sh`) has documentation inline indicating:
-   - What the service does / provides.
-   - What ports the service exposes.
-   - What other services it talks to (see environment variables).
-
-## Service system requirements
-
-Every service (`deploy-*.sh`) documents inline what the system requirements are (CPU/Memory/Disk allocation). For example, [the frontend service](https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/f01b97a397138dd76e5f5ed45b2574b9a2e70cd1/deploy-frontend.sh#L6-L9).
-
-## Scaling / replicas
-
-To scale the cluster deployment, you will need to:
-
-1. Deploy more instances of `gitserver`, `indexed-search`, `searcher` and `symbols` services as desired.
-2. [Configure the `frontend` and `frontend-internal` to communicate with the new instances.](https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/f01b97a397138dd76e5f5ed45b2574b9a2e70cd1/deploy-frontend.sh#L31-L34)
-3. You're done! You do not need to configure or restart any other services.
-
-## Metrics and Tracing
-
-This deployment comes with metrics and tracing built-in. See [metrics and tracing](metrics-and-tracing.md) for details.
-
-## Configuring HTTPS
-
-See https://docs.sourcegraph.com/admin/nginx#nginx-for-other-sourcegraph-clusters-e-g-pure-docker
-
-## Configuring SSH cloning
-
-1. Ensure your **Site admin** > **External services** code host configuration is configured to use SSH. For example, by setting the [`gitURLType`](https://docs.sourcegraph.com/admin/site_config/all#giturltype-string-enum) field to `"ssh"`. You may still need to configure an access token or other codehost authentication method in order for Sourcegraph to discover your repositories.
-
-Alternatively, you may use the `OTHER` codehost type under **External services**, which allows you to directly specify Git repository URLs for cloning.
-
-2. Provide your `gitserver` instances with your SSH / Git configuration (usually just `.ssh/id_rsa`, `.ssh/id_rsa.pub`, and `.ssh/known_hosts` -- but you can also provide other files like `.netrc`, `.gitconfig`, etc. if needed) by mounting it into the `sourcegraph` users home directory in the `gitserver` containers. For example, by adding the following flag:
-
-```
--v ~/my-sourcegraph-ssh-config/.ssh:/home/sourcegraph/.ssh
-```
-
-All future Git cloning operations will use the credentials configured there.
-
-If you wish, you can test that cloning with your configuration is working by performing the clone in a gitserver container shell, e.g. first acquire shell access:
-
-```
-$ docker exec -it gitserver-0 sh
-```
+## Questions & Issues
 
-Then try cloning the repository:
+[Open an issue (in sourcegraph/sourcegraph)](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=deploy-sourcegraph&template=deploy-sourcegraph.md&title=%5Bdeploy-sourcegraph%5D) or contact us (support@sourcegraph.com), we are happy to help!
 
-```
-$ git clone ssh://git@myhost.com/my/repo /tmp/my-repo
-```
-## Questions & Issues
+## Pure-Docker Sourcegraph cluster deployment reference
 
-[Open an issue (in sourcegraph/sourcegraph)](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=deploy-sourcegraph-docker&template=deploy-sourcegraph-docker.md&title=%5Bdeploy-sourcegraph-docker%5D) or contact us (support@sourcegraph.com), we are happy to help!
+What if your organization wants a multi-machine deployment without using Kubernetes?
+What if you use a different container management platform, for example?
+Anyone using a container management platform other than Kubernetes (Netflix's [Titus](https://netflix.github.io/titus/), Apache's [Mesos](http://mesos.apache.org/documentation/latest/docker-containerizer/), etc.) can use our [Pure-Docker Sourcegraph cluster deployment reference](./pure-docker/README.md) to deploy Sourcegraph.

RELEASING.md

@@ -1,5 +1,8 @@
 # Release guide
 
+**This is the guide for releasing the [Pure-Docker Sourcegraph deployment reference](./pure-docker/README.md).**
+The docker-compose release is done entirely via the [Sourcegraph release tool](https://about.sourcegraph.com/handbook/engineering/distribution/tools/release).
+
 ## Branching/tagging scheme
 
 Just like deploy-sourcegraph, we use version branches and version tags. We _additionally_ have a second set which is customer-replication branches and version tags:
@@ -68,10 +71,10 @@ Pretend `3.8` was the last version of pure-docker release (look for the latest `
 
 ```sh
 # Checkout the current pure-docker release branch
-git checkout 3.8-customer-replica 
+git checkout 3.8-customer-replica
 
 # Create the new pure-docker release branch
-git checkout -B 3.9-customer-replica 
+git checkout -B 3.9-customer-replica
 
 # Merge the publish-3.9 branch, which will have been created by the release tool, into the pure-docker release branch.
 git merge publish-3.9
@@ -95,9 +98,15 @@ At this point you should evaluate the `git status` output as well as all the cha
 During this process you will run into two merge conflicts:
 
 - Do not commit: `deploy-caddy.sh` or changes related to it, as `deploy-apache.sh` is used here.
-- Do not commit: changes to `deploy-pgsql.sh`, as Postgres 9.6 is used here.
+- The prettier ignore should look like this:
+
+```
+.github/
+*.md
+rootfs/etc/docker/daemon.json
+```
 
-4. Push the changes to github
+1. Push the changes to github
 ```shell
 git push --set-upstream origin 3.9-customer-replica
 ```

docker-compose/README.md

@@ -1,26 +1,5 @@
-# Sourcegraph for docker-compose
+# Sourcegraph with Docker Compose deployment reference
 
-This folder contains a Sourcegraph docker-compose deployment.
+This directory contains the Sourcegraph with Docker Compose deployment reference.
 
-## Deploying
-
-Simply clone the repository and `docker-compose up -d` to deploy Sourcegraph:
-
-```sh
-git clone https://github.com/sourcegraph/deploy-sourcegraph-docker
-cd deploy-sourcegraph-docker/docker-compose/
-git checkout v3.21.2
-docker-compose up -d
-```
-
-Sourcegraph will then run in the background and across server restarts.
-
-Notes:
-
-- The `docker-compose.yaml` file currently depends on configuration files which live in the repository, as such you must have the repository cloned onto your server.
-- Data for all services will be stored as docker volumes.
-- Use `docker ps` to inspect the Sourcegraph containers, and `docker-compose down` to teardown the deployment.
-
-## Upgrading
-
-Always refer to https://docs.sourcegraph.com/admin/updates/docker_compose prior to upgrading, as it will document any manual steps you may need to take in order to upgrade smoothly.
+To learn more about deploying, configuring, and upgrading a Sourcegraph with Docker Compose installation, please refer to our documentation: [Sourcegraph with Docker Compose](https://docs.sourcegraph.com/admin/install/docker-compose)

docker-compose/db-only-migrate.docker-compose.yaml

@@ -12,7 +12,7 @@ services:
   #
   pgsql:
     container_name: pgsql
-    image: 'index.docker.io/sourcegraph/postgres-12.6-alpine:3.30.0@sha256:439246675646322cf6b22a277ab86d790287bc4c532e2a2f4a4e0538d06cd206'
+    image: 'index.docker.io/sourcegraph/postgres-12.6:95244_2021-05-06_2c1f77e@sha256:35040317490324a15e1259c9023a726eb27c694530f6f8877e87d337c7b97778'
     cpus: 4
     mem_limit: '2g'
     healthcheck:

docker-compose/docker-compose.yaml

@@ -69,7 +69,7 @@ services:
   # service.
   sourcegraph-frontend-0:
     container_name: sourcegraph-frontend-0
-    image: 'index.docker.io/sourcegraph/frontend:insiders@sha256:423d9d7f14107339792b4ec5e3610b57d01dce0a390e30bbfefda1c163587a56'
+    image: 'index.docker.io/sourcegraph/frontend:insiders@sha256:07206787b154da77d133333a507bdd26df7fb6f17b696bcf1c628ac8f69a4236'
     cpus: 4
     mem_limit: '8g'
     environment:
@@ -113,7 +113,7 @@ services:
   #
   sourcegraph-frontend-internal:
     container_name: sourcegraph-frontend-internal
-    image: 'index.docker.io/sourcegraph/frontend:insiders@sha256:423d9d7f14107339792b4ec5e3610b57d01dce0a390e30bbfefda1c163587a56'
+    image: 'index.docker.io/sourcegraph/frontend:insiders@sha256:07206787b154da77d133333a507bdd26df7fb6f17b696bcf1c628ac8f69a4236'
     cpus: 4
     mem_limit: '8g'
     environment:
@@ -150,6 +150,7 @@ services:
       codeintel-db:
         condition: service_healthy
 
+
   # Description: Stores clones of repositories to perform Git operations.
   #
   # Disk: 200GB / persistent SSD
@@ -158,7 +159,7 @@ services:
   #
   gitserver-0:
     container_name: gitserver-0
-    image: 'index.docker.io/sourcegraph/gitserver:insiders@sha256:320febe0e9f15ab0d25bfbbfec83caec0757a5ed26d600548c678b969f66f2aa'
+    image: 'index.docker.io/sourcegraph/gitserver:insiders@sha256:2c2fb7ba482518481be45a17b0ac81698c3062efa2e4e2133ab928ab031a43bc'
     cpus: 4
     mem_limit: '8g'
     environment:
@@ -228,7 +229,7 @@ services:
   #
   searcher-0:
     container_name: searcher-0
-    image: 'index.docker.io/sourcegraph/searcher:insiders@sha256:6e0023081f993c654ad4ea7a63b7e9d307448c5533c836f0b3b5243db959fa56'
+    image: 'index.docker.io/sourcegraph/searcher:insiders@sha256:8c5751c7a499cae8ba0ad772b06a02d6a8270021946fde71937480dd42ac64ac'
     cpus: 2
     mem_limit: '2g'
     environment:
@@ -256,7 +257,7 @@ services:
   #
   github-proxy:
     container_name: github-proxy
-    image: 'index.docker.io/sourcegraph/github-proxy:insiders@sha256:e8148f33b62479bf563e05ad0923f84ce9be38b45043889194076eff4bed4e4b'
+    image: 'index.docker.io/sourcegraph/github-proxy:insiders@sha256:1e6c71f78d4729208bf2016977a18b7fa9a4856bbb111546c66bff49f914b54b'
     cpus: 1
     mem_limit: '1g'
     environment:
@@ -274,7 +275,7 @@ services:
   #
   precise-code-intel-worker:
     container_name: precise-code-intel-worker
-    image: 'index.docker.io/sourcegraph/precise-code-intel-worker:insiders@sha256:ca92df3fb60f09c634227ded0f8f1d8549fbca49c5e9976a71d15a48916052ca'
+    image: 'index.docker.io/sourcegraph/precise-code-intel-worker:insiders@sha256:62992ccbd790f72e59bb03a79d13bcdc46bdf5680c01e84fb44cc51a9d484f8e'
     cpus: 2
     mem_limit: '4g'
     environment:
@@ -300,7 +301,7 @@ services:
   #
   query-runner:
     container_name: query-runner
-    image: 'index.docker.io/sourcegraph/query-runner:insiders@sha256:c3d9a10bbf0697b330025fa5e5cd8571e358fd23f48cbbbcf47e4dc0a606b9c2'
+    image: 'index.docker.io/sourcegraph/query-runner:insiders@sha256:59721e8267aa7ee95822c8609e9bba1ad1f6ebebf19b27203fe31905f5169278'
     cpus: 1
     mem_limit: '1g'
     environment:
@@ -319,7 +320,7 @@ services:
   #
   repo-updater:
     container_name: repo-updater
-    image: 'index.docker.io/sourcegraph/repo-updater:insiders@sha256:fbcdc6735331693ce682ad9553240cb217d24e3c40234535a8bdb528ffdf2295'
+    image: 'index.docker.io/sourcegraph/repo-updater:insiders@sha256:2b2b3f34bbb6d6dd045d976843337aaa29057b44637eddc352c7c8ba9da62c2c'
     cpus: 4
     mem_limit: '4g'
     environment:
@@ -341,7 +342,7 @@ services:
   #
   worker:
     container_name: worker
-    image: 'index.docker.io/sourcegraph/worker:insiders@sha256:293bc7804869b66714002b92a84d5c5b9690602bf2d83474761074c6eae8cfc6'
+    image: 'index.docker.io/sourcegraph/worker:insiders@sha256:b6bfaf41e075f1e9c342a39881f578a344c151c70dcf93099ebf8a82ad0011da'
     cpus: 4
     mem_limit: '4g'
     environment:
@@ -383,7 +384,7 @@ services:
   #
   symbols-0:
     container_name: symbols-0
-    image: 'index.docker.io/sourcegraph/symbols:insiders@sha256:0eac2d380b05b288f7bec37324870e1cc425d92f290420edd9714eea81f592e1'
+    image: 'index.docker.io/sourcegraph/symbols:insiders@sha256:bdd8ee612b0291859874af64fca88d14a425be7670d8f49a06fc360fe7a433f7'
     cpus: 2
     mem_limit: '4g'
     environment:
@@ -458,7 +459,7 @@ services:
   #
   cadvisor:
     container_name: cadvisor
-    image: 'index.docker.io/sourcegraph/cadvisor:insiders@sha256:6b7f3d4a6ab00444f7df65e74d6d552462280b6b73e605827b5a759877f6a1dd'
+    image: 'index.docker.io/sourcegraph/cadvisor:insiders@sha256:ae163e3912bbbb1e2abd81581b81dc36b8ee9b3264805a7bde8bfb9f47981253'
     cpus: 1
     mem_limit: '1g'
     volumes:
@@ -485,7 +486,7 @@ services:
   #
   jaeger:
     container_name: jaeger
-    image: 'index.docker.io/sourcegraph/jaeger-all-in-one:insiders@sha256:bbeea94c57f3b395858e39d39f0df9beb5d2b93adad93f030f8d8184da41b11f'
+    image: 'index.docker.io/sourcegraph/jaeger-all-in-one:insiders@sha256:5ac56c83b2284a19cc736fe66cf11facaa084a28417382b35773ff1ab87fe166'
     cpus: 0.5
     mem_limit: '512m'
     ports:
@@ -510,7 +511,7 @@ services:
   #
   pgsql:
     container_name: pgsql
-    image: 'index.docker.io/sourcegraph/postgres-12.6-alpine:3.30.0@sha256:439246675646322cf6b22a277ab86d790287bc4c532e2a2f4a4e0538d06cd206'
+    image: 'index.docker.io/sourcegraph/postgres-12.6:95244_2021-05-06_2c1f77e@sha256:35040317490324a15e1259c9023a726eb27c694530f6f8877e87d337c7b97778'
     cpus: 4
     mem_limit: '2g'
     healthcheck:
@@ -524,6 +525,7 @@ services:
     networks:
       - sourcegraph
     restart: always
+    stop_grace_period: 120s
 
   # Description: PostgreSQL database for code intelligence data.
   #
@@ -548,6 +550,7 @@ services:
     networks:
       - sourcegraph
     restart: always
+    stop_grace_period: 120s
 
   # Description: TimescaleDB time-series database for code insights data.
   #
@@ -562,17 +565,18 @@ services:
   # would be bad but it can be rebuilt given enough time.)
   codeinsights-db:
     container_name: codeinsights-db
-    image: 'index.docker.io/sourcegraph/codeinsights-db:insiders@sha256:75be25ec13a1a2706adefff68c8101a58d692e8d36ebac3e43f33b9304e5b8c7'
+    image: "index.docker.io/sourcegraph/codeinsights-db:insiders@sha256:75be25ec13a1a2706adefff68c8101a58d692e8d36ebac3e43f33b9304e5b8c7"
     cpus: 4
-    mem_limit: '2g'
+    mem_limit: "2g"
     environment:
       - POSTGRES_PASSWORD=password
       - PGDATA=/var/lib/postgresql/data/pgdata
     volumes:
-      - 'codeinsights-db:/var/lib/postgresql/data/'
+      - "codeinsights-db:/var/lib/postgresql/data/"
     networks:
       - sourcegraph
     restart: always
+    stop_grace_period: 120s
 
   # Description: MinIO for storing LSIF uploads.
   #

go.mod

@@ -4,5 +4,5 @@ go 1.15
 
 require (
 	github.com/slimsag/update-docker-tags v0.7.0
-	github.com/sourcegraph/sourcegraph/enterprise/dev/ci/images v0.0.0-20210723035421-9fd81850e251
+	github.com/sourcegraph/sourcegraph/enterprise/dev/ci/images v0.0.0-20210731040113-a2d19524cb65
 )