Compare commits
1 commit
f4639b0fae
...
6ad24bf848
Author | SHA1 | Date | |
---|---|---|---|
Reynir Björnsson | 6ad24bf848 |
105
README.md
105
README.md
|
@ -1,105 +0,0 @@
|
||||||
# data.coop infrastructure
|
|
||||||
|
|
||||||
This repository contains the code used to deploy data.coop's services
|
|
||||||
and websites. We use Ansible to encode our infrastructure setup. Only
|
|
||||||
the association's administrators have access to deploy the services.
|
|
||||||
|
|
||||||
## Deploying
|
|
||||||
|
|
||||||
To deploy the services, the included `deploy.sh` script can be used. The
|
|
||||||
Ansible playbook uses two custom-made roles (in the `roles/` directory):
|
|
||||||
|
|
||||||
- `ubuntu_base` - used to configure the host itself and install the
|
|
||||||
necessary packages
|
|
||||||
- `docker` - used to deploy our services and websites with Docker
|
|
||||||
containers
|
|
||||||
|
|
||||||
The script has options to deploy only one of the roles. Select services
|
|
||||||
only can also be specified. By default, the script deploys everything.
|
|
||||||
|
|
||||||
Here is a summary of the options that can be used with the script:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# deploy everything
|
|
||||||
./deploy.sh
|
|
||||||
|
|
||||||
# deploy the ubuntu_base role only
|
|
||||||
./deploy.sh base
|
|
||||||
|
|
||||||
# deploy the docker role only
|
|
||||||
./deploy.sh services
|
|
||||||
|
|
||||||
# deploy SINGLE_SERVICE Docker service only
|
|
||||||
./deploy.sh services SINGLE_SERVICE
|
|
||||||
```
|
|
||||||
|
|
||||||
`SINGLE_SERVICE` should match one of the service names in the `services`
|
|
||||||
dictionary in `roles/docker/defaults/main.yml` (e.g. `gitea` or
|
|
||||||
`data_coop_website`).
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
In order for us to be able to test our setup locally, we use Vagrant to
|
|
||||||
deploy the services in a virtual machine. To do this, Vagrant and
|
|
||||||
VirtualBox must both be installed on the development machine. Then, the
|
|
||||||
services can be deployed locally by using the `vagrant` command-line
|
|
||||||
tool. The working directory needs to be the root of the repository for
|
|
||||||
this to work properly.
|
|
||||||
|
|
||||||
> Note: As our secrets are contained in an Ansible Vault file, only the
|
|
||||||
> administrators have the ability to run the deployment in Vagrant.
|
|
||||||
> However, one could replace the vault file for testing purposes.
|
|
||||||
|
|
||||||
Here is a summary of the commands that are available with the `vagrant`
|
|
||||||
command-line tool:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# Create and provision the VM
|
|
||||||
vagrant up
|
|
||||||
|
|
||||||
# Re-provision the VM
|
|
||||||
vagrant provision
|
|
||||||
|
|
||||||
# SSH into the VM
|
|
||||||
vagrant ssh
|
|
||||||
|
|
||||||
# Power down the VM
|
|
||||||
vagrant halt
|
|
||||||
|
|
||||||
# Power down and delete the VM
|
|
||||||
vagrant destroy
|
|
||||||
```
|
|
||||||
|
|
||||||
The `vagrant` command-line tool does not support supplying extra
|
|
||||||
variables to Ansible on runtime, so to be able to deploy only parts of
|
|
||||||
the Ansible playbook to Vagrant, the `deploy.sh` script can be used with
|
|
||||||
the `--vagrant` flag. Here are some examples:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# deploy the ubuntu_base role only in the Vagrant VM
|
|
||||||
./deploy.sh --vagrant base
|
|
||||||
|
|
||||||
# deploy SINGLE_SERVICE Docker service only in the Vagrant VM
|
|
||||||
./deploy.sh --vagrant services SINGLE_SERVICE
|
|
||||||
```
|
|
||||||
|
|
||||||
Note that the `--vagrant` flag should be the first argument when using
|
|
||||||
the script.
|
|
||||||
|
|
||||||
## Contributing
|
|
||||||
|
|
||||||
If you want to contribute, you can fork the repository and submit a pull
|
|
||||||
request. We use a pre-commit hook for linting the YAML files before
|
|
||||||
every commit, so please use that. To initialize pre-commit, you need to
|
|
||||||
have Python and GNU make installed. Then, just run the following shell
|
|
||||||
command:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
make init
|
|
||||||
```
|
|
||||||
|
|
||||||
## Nice tools
|
|
||||||
|
|
||||||
- [J2Live](https://j2live.ttl255.com/): A live Jinja2 parser, nice to
|
|
||||||
test out filters
|
|
||||||
|
|
10
deploy.sh
10
deploy.sh
|
@ -1,13 +1,5 @@
|
||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
|
|
||||||
usage () {
|
|
||||||
{
|
|
||||||
echo "Usage: $0 [--vagrant]"
|
|
||||||
echo "Usage: $0 [--vagrant] base"
|
|
||||||
echo "Usage: $0 [--vagrant] services [SERVICE]"
|
|
||||||
} >&2
|
|
||||||
}
|
|
||||||
|
|
||||||
BASE_CMD="ansible-playbook playbook.yml --ask-vault-pass"
|
BASE_CMD="ansible-playbook playbook.yml --ask-vault-pass"
|
||||||
|
|
||||||
if [ "$1" = "--vagrant" ]; then
|
if [ "$1" = "--vagrant" ]; then
|
||||||
|
@ -38,7 +30,7 @@ else
|
||||||
$BASE_CMD --tags base_only
|
$BASE_CMD --tags base_only
|
||||||
;;
|
;;
|
||||||
*)
|
*)
|
||||||
usage
|
echo >&2 "Command \"$1\" not found!"
|
||||||
exit 1
|
exit 1
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
|
@ -1,20 +0,0 @@
|
||||||
# DB Version: 14
|
|
||||||
# OS Type: linux
|
|
||||||
# DB Type: oltp
|
|
||||||
# Total Memory (RAM): 16 GB
|
|
||||||
# Connections num: 300
|
|
||||||
# Data Storage: hdd
|
|
||||||
|
|
||||||
listen_addresses = '*'
|
|
||||||
max_connections = 300
|
|
||||||
shared_buffers = 4GB
|
|
||||||
effective_cache_size = 12GB
|
|
||||||
maintenance_work_mem = 1GB
|
|
||||||
checkpoint_completion_target = 0.9
|
|
||||||
wal_buffers = 16MB
|
|
||||||
default_statistics_target = 100
|
|
||||||
random_page_cost = 4
|
|
||||||
effective_io_concurrency = 2
|
|
||||||
work_mem = 6990kB
|
|
||||||
min_wal_size = 2GB
|
|
||||||
max_wal_size = 8GB
|
|
|
@ -16,40 +16,16 @@
|
||||||
src: files/configs/mastodon/env_file.j2
|
src: files/configs/mastodon/env_file.j2
|
||||||
dest: "{{ services.mastodon.volume_folder }}/env_file"
|
dest: "{{ services.mastodon.volume_folder }}/env_file"
|
||||||
|
|
||||||
- name: Upload vhost config for root domain
|
- name: upload vhost config for root domain
|
||||||
template:
|
template:
|
||||||
src: files/configs/mastodon/vhost-mastodon
|
src: files/configs/mastodon/vhost-mastodon
|
||||||
dest: "{{ services.nginx_proxy.volume_folder }}/vhost/{{ services.mastodon.domain }}"
|
dest: "{{ services.nginx_proxy.volume_folder }}/vhost/{{ services.mastodon.domain }}"
|
||||||
|
|
||||||
- name: Copy postgresql config
|
|
||||||
copy:
|
|
||||||
src: files/configs/mastodon/postgresql.conf
|
|
||||||
dest: "{{ services.mastodon.volume_folder }}/postgres_data/postgresql.conf"
|
|
||||||
|
|
||||||
- name: set up mastodon
|
- name: set up mastodon
|
||||||
docker_compose:
|
docker_compose:
|
||||||
project_name: mastodon
|
project_name: mastodon
|
||||||
pull: yes
|
pull: yes
|
||||||
definition:
|
definition:
|
||||||
|
|
||||||
x-sidekiq: &sidekiq
|
|
||||||
image: "tootsuite/mastodon:{{ services.mastodon.version }}"
|
|
||||||
restart: always
|
|
||||||
env_file: "{{ services.mastodon.volume_folder }}/env_file"
|
|
||||||
depends_on:
|
|
||||||
db:
|
|
||||||
condition: "service_healthy"
|
|
||||||
redis:
|
|
||||||
condition: "service_healthy"
|
|
||||||
networks:
|
|
||||||
- postfix
|
|
||||||
- external_services
|
|
||||||
- internal_network
|
|
||||||
volumes:
|
|
||||||
- "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
|
|
||||||
healthcheck:
|
|
||||||
test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
|
|
||||||
|
|
||||||
version: '3'
|
version: '3'
|
||||||
services:
|
services:
|
||||||
db:
|
db:
|
||||||
|
@ -94,8 +70,6 @@
|
||||||
volumes:
|
volumes:
|
||||||
- "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
|
- "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
|
||||||
environment:
|
environment:
|
||||||
MAX_THREADS: 10
|
|
||||||
WEB_CONCURRENCY: 3
|
|
||||||
VIRTUAL_HOST: "{{ services.mastodon.domain }}"
|
VIRTUAL_HOST: "{{ services.mastodon.domain }}"
|
||||||
VIRTUAL_PORT: "3000"
|
VIRTUAL_PORT: "3000"
|
||||||
VIRTUAL_PATH: "/"
|
VIRTUAL_PATH: "/"
|
||||||
|
@ -121,59 +95,30 @@
|
||||||
redis:
|
redis:
|
||||||
condition: "service_healthy"
|
condition: "service_healthy"
|
||||||
environment:
|
environment:
|
||||||
DB_POOL: 15
|
|
||||||
VIRTUAL_HOST: "{{ services.mastodon.domain }}"
|
VIRTUAL_HOST: "{{ services.mastodon.domain }}"
|
||||||
VIRTUAL_PORT: "4000"
|
VIRTUAL_PORT: "4000"
|
||||||
VIRTUAL_PATH: "/api/v1/streaming"
|
VIRTUAL_PATH: "/api/v1/streaming"
|
||||||
|
|
||||||
# sidekiq-default-push-pull: DB_POOL = 25, -c 25 for 25 connections
|
sidekiq:
|
||||||
sidekiq-default-push-pull:
|
image: "tootsuite/mastodon:{{ services.mastodon.version }}"
|
||||||
<<: *sidekiq
|
restart: always
|
||||||
command: bundle exec sidekiq -c 25 -q default -q push -q pull
|
env_file: "{{ services.mastodon.volume_folder }}/env_file"
|
||||||
|
command: bundle exec sidekiq -c 32
|
||||||
environment:
|
environment:
|
||||||
DB_POOL: 25
|
DB_POOL: 32
|
||||||
|
depends_on:
|
||||||
# sidekiq-default-pull-push: DB_POOL = 25, -c 25 for 25 connections
|
db:
|
||||||
sidekiq-default-pull-push:
|
condition: "service_healthy"
|
||||||
<<: *sidekiq
|
redis:
|
||||||
command: bundle exec sidekiq -c 25 -q default -q pull -q push
|
condition: "service_healthy"
|
||||||
environment:
|
networks:
|
||||||
DB_POOL: 25
|
- postfix
|
||||||
|
- external_services
|
||||||
# sidekiq-pull-default-push: DB_POOL = 25, -c 25 for 25 connections
|
- internal_network
|
||||||
sidekiq-pull-default-push:
|
volumes:
|
||||||
<<: *sidekiq
|
- "{{ services.mastodon.volume_folder }}/mastodon_data:/mastodon/public/system"
|
||||||
command: bundle exec sidekiq -c 25 -q pull -q default -q push
|
healthcheck:
|
||||||
environment:
|
test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
|
||||||
DB_POOL: 25
|
|
||||||
|
|
||||||
# sidekiq-push-default-pull: DB_POOL = 25, -c 25 for 25 connections
|
|
||||||
sidekiq-push-default-pull:
|
|
||||||
<<: *sidekiq
|
|
||||||
command: bundle exec sidekiq -c 25 -q push -q default -q pull
|
|
||||||
environment:
|
|
||||||
DB_POOL: 25
|
|
||||||
|
|
||||||
# sidekiq-push-scheduler: DB_POOL = 5, -c 5 for 5 connections
|
|
||||||
sidekiq-push-scheduler:
|
|
||||||
<<: *sidekiq
|
|
||||||
command: bundle exec sidekiq -c 5 -q push -q scheduler
|
|
||||||
environment:
|
|
||||||
DB_POOL: 5
|
|
||||||
|
|
||||||
# sidekiq-push-mailers: DB_POOL = 5, -c 5 for 5 connections
|
|
||||||
sidekiq-push-mailers:
|
|
||||||
<<: *sidekiq
|
|
||||||
command: bundle exec sidekiq -c 5 -q push -q mailers
|
|
||||||
environment:
|
|
||||||
DB_POOL: 5
|
|
||||||
|
|
||||||
# sidekiq-push-ingress: DB_POOL = 10, -c 10 for 10 connections
|
|
||||||
sidekiq-push-ingress:
|
|
||||||
<<: *sidekiq
|
|
||||||
command: bundle exec sidekiq -c 10 -q push -q ingress
|
|
||||||
environment:
|
|
||||||
DB_POOL: 10
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
external_services:
|
external_services:
|
||||||
|
|
Loading…
Reference in a new issue