social.data.coop: return 200 OK if datadog

It's monitoring setup by someone else that we didn't opt into.

README.md

@@ -0,0 +1,105 @@
+# data.coop infrastructure
+
+This repository contains the code used to deploy data.coop's services
+and websites. We use Ansible to encode our infrastructure setup. Only
+the association's administrators have access to deploy the services.
+
+## Deploying
+
+To deploy the services, the included `deploy.sh` script can be used. The
+Ansible playbook uses two custom-made roles (in the `roles/` directory):
+
+- `ubuntu_base` - used to configure the host itself and install the
+  necessary packages
+- `docker` - used to deploy our services and websites with Docker
+  containers
+
+The script has options to deploy only one of the roles. Select services
+only can also be specified. By default, the script deploys everything.
+
+Here is a summary of the options that can be used with the script:
+
+```sh
+# deploy everything
+./deploy.sh
+
+# deploy the ubuntu_base role only
+./deploy.sh base
+
+# deploy the docker role only
+./deploy.sh services
+
+# deploy SINGLE_SERVICE Docker service only
+./deploy.sh services SINGLE_SERVICE
+```
+
+`SINGLE_SERVICE` should match one of the service names in the `services`
+dictionary in `roles/docker/defaults/main.yml` (e.g. `gitea` or
+`data_coop_website`).
+
+## Testing
+
+In order for us to be able to test our setup locally, we use Vagrant to
+deploy the services in a virtual machine. To do this, Vagrant and
+VirtualBox must both be installed on the development machine. Then, the
+services can be deployed locally by using the `vagrant` command-line
+tool. The working directory needs to be the root of the repository for
+this to work properly.
+
+> Note: As our secrets are contained in an Ansible Vault file, only the
+> administrators have the ability to run the deployment in Vagrant.
+> However, one could replace the vault file for testing purposes.
+
+Here is a summary of the commands that are available with the `vagrant`
+command-line tool:
+
+```sh
+# Create and provision the VM
+vagrant up
+
+# Re-provision the VM
+vagrant provision
+
+# SSH into the VM
+vagrant ssh
+
+# Power down the VM
+vagrant halt
+
+# Power down and delete the VM
+vagrant destroy
+```
+
+The `vagrant` command-line tool does not support supplying extra
+variables to Ansible on runtime, so to be able to deploy only parts of
+the Ansible playbook to Vagrant, the `deploy.sh` script can be used with
+the `--vagrant` flag. Here are some examples:
+
+```sh
+# deploy the ubuntu_base role only in the Vagrant VM
+./deploy.sh --vagrant base
+
+# deploy SINGLE_SERVICE Docker service only in the Vagrant VM
+./deploy.sh --vagrant services SINGLE_SERVICE
+```
+
+Note that the `--vagrant` flag should be the first argument when using
+the script.
+
+## Contributing
+
+If you want to contribute, you can fork the repository and submit a pull
+request. We use a pre-commit hook for linting the YAML files before
+every commit, so please use that. To initialize pre-commit, you need to
+have Python and GNU make installed. Then, just run the following shell
+command:
+
+```sh
+make init
+```
+
+## Nice tools
+
+- [J2Live](https://j2live.ttl255.com/): A live Jinja2 parser, nice to
+  test out filters
+

deploy.sh

@@ -1,5 +1,13 @@
 #!/bin/sh
 
+usage () {
+    {
+        echo "Usage: $0 [--vagrant]"
+        echo "Usage: $0 [--vagrant] base"
+        echo "Usage: $0 [--vagrant] services [SERVICE]"
+    } >&2
+}
+
 BASE_CMD="ansible-playbook playbook.yml --ask-vault-pass"
 
 if [ "$1" = "--vagrant" ]; then
@@ -30,7 +38,7 @@ else
             $BASE_CMD --tags base_only
         ;;
         *)
-            echo >&2 "Command \"$1\" not found!"
+            usage
             exit 1
         ;;
     esac

roles/docker/files/configs/mastodon/postgresql.conf

@@ -0,0 +1,20 @@
+# DB Version: 14
+# OS Type: linux
+# DB Type: oltp
+# Total Memory (RAM): 16 GB
+# Connections num: 300
+# Data Storage: hdd
+
+listen_addresses = '*'
+max_connections = 300
+shared_buffers = 4GB
+effective_cache_size = 12GB
+maintenance_work_mem = 1GB
+checkpoint_completion_target = 0.9
+wal_buffers = 16MB
+default_statistics_target = 100
+random_page_cost = 4
+effective_io_concurrency = 2
+work_mem = 6990kB
+min_wal_size = 2GB
+max_wal_size = 8GB

roles/docker/tasks/services/mastodon.yml

@@ -16,16 +16,40 @@
     src: files/configs/mastodon/env_file.j2
     dest: "{{ services.mastodon.volume_folder }}/env_file"
 
-- name: upload vhost config for root domain
+- name: Upload vhost config for root domain
   template:
     src: files/configs/mastodon/vhost-mastodon
     dest: "{{ services.nginx_proxy.volume_folder }}/vhost/{{ services.mastodon.domain }}"
 
+- name: Copy postgresql config
+  copy:
+    src: files/configs/mastodon/postgresql.conf
+    dest: "{{ services.mastodon.volume_folder }}/postgres_data/postgresql.conf"
+
 - name: set up mastodon
   docker_compose:
     project_name: mastodon
     pull: yes
     definition:
+
+      x-sidekiq: &sidekiq
+        image: "tootsuite/mastodon:{{ services.mastodon.version }}"
+        restart: always
+        env_file: "{{ services.mastodon.volume_folder }}/env_file"
+        depends_on:
+          db:
+            condition: "service_healthy"
+          redis:
+            condition: "service_healthy"
+        networks:
+          - postfix
+          - external_services
+          - internal_network
+        volumes:
+          - "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
+        healthcheck:
+          test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
+
       version: '3'
       services:
         db:
@@ -70,6 +94,8 @@
           volumes:
             - "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
           environment:
+            MAX_THREADS: 10
+            WEB_CONCURRENCY: 3
             VIRTUAL_HOST: "{{ services.mastodon.domain }}"
             VIRTUAL_PORT: "3000"
             VIRTUAL_PATH: "/"
@@ -95,30 +121,59 @@
             redis:
               condition: "service_healthy"
           environment:
+            DB_POOL: 15
             VIRTUAL_HOST: "{{ services.mastodon.domain }}"
             VIRTUAL_PORT: "4000"
             VIRTUAL_PATH: "/api/v1/streaming"
 
-        sidekiq:
-          image: "tootsuite/mastodon:{{ services.mastodon.version }}"
-          restart: always
-          env_file: "{{ services.mastodon.volume_folder }}/env_file"
-          command: bundle exec sidekiq -c 32
+        # sidekiq-default-push-pull: DB_POOL = 25, -c 25 for 25 connections
+        sidekiq-default-push-pull:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 25 -q default -q push -q pull
           environment:
-            DB_POOL: 32
-          depends_on:
-            db:
-              condition: "service_healthy"
-            redis:
-              condition: "service_healthy"
-          networks:
-            - postfix
-            - external_services
-            - internal_network
-          volumes:
-            - "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
-          healthcheck:
-            test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
+            DB_POOL: 25
+
+        # sidekiq-default-pull-push: DB_POOL = 25, -c 25 for 25 connections
+        sidekiq-default-pull-push:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 25 -q default -q pull -q push
+          environment:
+            DB_POOL: 25
+
+        # sidekiq-pull-default-push: DB_POOL = 25, -c 25 for 25 connections
+        sidekiq-pull-default-push:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 25 -q pull -q default -q push
+          environment:
+            DB_POOL: 25
+
+        # sidekiq-push-default-pull: DB_POOL = 25, -c 25 for 25 connections
+        sidekiq-push-default-pull:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 25 -q push -q default -q pull
+          environment:
+            DB_POOL: 25
+
+        # sidekiq-push-scheduler: DB_POOL = 5, -c 5 for 5 connections
+        sidekiq-push-scheduler:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 5 -q push -q scheduler
+          environment:
+            DB_POOL: 5
+
+        # sidekiq-push-mailers: DB_POOL = 5, -c 5 for 5 connections
+        sidekiq-push-mailers:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 5 -q push -q mailers
+          environment:
+            DB_POOL: 5
+
+        # sidekiq-push-ingress: DB_POOL = 10, -c 10 for 10 connections
+        sidekiq-push-ingress:
+          <<: *sidekiq
+          command: bundle exec sidekiq -c 10 -q push -q ingress
+          environment:
+            DB_POOL: 10
 
       networks:
         external_services: