From be783a68033872bf8584179c662d947858714755 Mon Sep 17 00:00:00 2001 From: Hannes Mehnert Date: Sun, 27 Jan 2019 19:15:38 +0100 Subject: [PATCH] README: updates, remove outdated instructions --- README.md | 242 +++++++++++++++--------------------------------------- 1 file changed, 67 insertions(+), 175 deletions(-) diff --git a/README.md b/README.md index 93a1c4a..e5b04e3 100644 --- a/README.md +++ b/README.md @@ -2,188 +2,80 @@ [![Build Status](https://travis-ci.org/hannesm/albatross.svg?branch=master)](https://travis-ci.org/hannesm/albatross) -A set of binaries to manage, provision, and deploy MirageOS unikernels. -Some daemons are supposed to run in the host system, communicating via Unix domain sockets: -- `vmmd`: privileged to create and destroy unikernels (also creates tap devices and attaches these to bridges) -- `vmmd_console`: reads the console output of unikernels (via a fifo passed from `vmmd`) +The goal of albatross is robust deployment of [MirageOS](https://mirage.io) +unikernels using [Solo5](https://github.com/solo5/solo5), including precise +error handling of failures. The code running under superuser privileges is +minimimal. Albatross is supposed to be run on a machine in the dom0, next to the +hypervisor. Albatross keeps track of unikernel resource usage (memory, CPUs, +bridges, block storage and active block devices). Policies restricting these +resources for administrative domains are available. Local and remote deployments +are supported, remote ones are authenticated and encrypted via a mutually +authenticated TLS connection using X.509 client certificates. Multi-tenancy +deployments are possible, tenants do not need any other access to the machine: +console output and statistics gathered by the host are accessible via TLS. +Albatross keeps the information of running unikernels persistently, and starts +these unikernels when the albatross daemon is started. This means that whenever +a unikernel was started, it keeps running until it crashes or an explicit +destroy command is issued. + +The administrative domain is similar to DNS: each unikernel has a name (e.g. +`foo.hello`), which consists of labels separated by dots. Policies and +access is done on a name basis - if access to `foo` is granted, `foo.hello`, +`foo.bar.hello`, etc. can be accessed, but not `bar` or `bar.hello`. + +## Components + +Albatross consists of a set of binaries. Several daemons, which communicate in a +request-response style over Unix domain sockets, are run in the host system: +- `vmmd`: privileged to create and destroy unikernels +- `vmmd_console`: reads the console output of unikernels - `vmmd_log`: event log -- `vmmd_stats`: statistics (`getrusage`, ifstat, BHyve debug counters) gathering -- `vmmd_tls`: opens a TCP socket which authenticates (TLS with client certificate) and proxies to local unix domain sockets -- `vmmd_tls_inetd`: authenticates (TLS with client certificate via stdin) and proxies commands carried by a client certificate, to be used by inetd -- `vmmd_influx`: reports statistics from stats to influx listener +- `vmmd_stats`: statistics gathering (rusage, ifstat, BHyve debug counters) +- `vmmd_tls`: remote deployment via TLS with client certificate, and proxies to local daemons +- `vmmd_tls_inetd`: remote deployment via TLS and inetd (alternative to `vmmd_tls`) +- `vmmd_influx`: statistic reporting from `vmmd_stats` to influx -Command-line applications for local and remote management are provided as well -- `vmmc_local`: executes a command locally via Unix domain sockets -- `vmmc_remote`: connects to `vmm_tls_endpoint` and executes command +The main daemon is the privileged `vmmd`, which supervises unikernels. It opens +a listening Unix domain socket, reads the persisted unikernel configuration, +starts these unikernels, and awaits commands. Access can be regulated by Unix +file permissions, only those users who can write to that socket can send +commands. + +`Vmmd_console` does not keep any persistent state, but a ring buffer of console +output from each unikernel. These messages can be retrieved by a client, as a +stream of messages (history, and whenever a new line is output, it is send to +the interested client). Each unikernel output can only be read by a single +client, to avoid amplification of traffic if lots of clients are connected. +`Vmmd` sends a message to `vmmd_console` whenever a new unikernel is started, +upon reception `Vmmd_console` opens and reads the fifo which the unikernel will +write their standard output to. + +`Vmmd_log` keeps a persistent event log for albatross, can be read by clients. + +`Vmmd_stats` gathers periodically statistics (memory, CPU, network, hypervisor) +from all running unikernels. + +`Vmmd_tls` and `vmmd_tls_inetd` listen on a TCP port, and proxy requests from +remote clients to the respective daemons described above. They enforce client +authentication, and use the commen names of the client certificate chain as +administrative domain. The policies are embedded in CA certificates, the command +is embedded in the leaf certificate. + +The following command-line applications for local and remote management are provided: +- `vmmc_local`: sends a command locally to the Unix domain sockets +- `vmmc_remote`: connects to a remote TLS endpoint and sends a command - `vmmp_request`: creates a certificate signing request containing a command -- `vmmp_ca`: certificate authority operations: sign, generate (and revoke) -- `vmmc_bistro`: command line utility to execute a command remotely: request, sign, remote (do not use in production, requires CA key on host) +- `vmmp_ca`: certificate authority operations: sign, generate, and revoke (NYI) +- `vmmc_bistro`: command line utility to execute a command remotely: request, sign, remote (do not use in production, requires CA key locally) -TODO: from here on, this documentation is not up to date. - -Please read [the (outdated) blog article](https://hannes.nqsb.io/Posts/VMM) for motivation -and an overview. - -The implementation uses explicit errors (no exceptions), and make mostly use of -the (blocking!) Bos library for operating system commands. A thin layer of Lwt -is used on top to (more gracefully) handle multiple connection, and to have a -watching thread (in `waitpid(2)`) for every virtual machine started by vmmd. +## Installation To install Albatross, run `opam pin add albatross https://github.com/hannesm/albatross`. -The following elaborates on how to get the software up and running, following by -provisioning and deploying some unikernels. There is a *server* (`SRV`) -component which needs six binaries: vmm_console, vmm_log, vmm_stats_lwt, vmmd, -solo5-hvt.none, solo5-hvt.net, solo5-hvt.block and solo5-hvt.block-net; a `CA` machine (which should be air-gapped, or -at least use some hardware token) for provisioning which needs vmm_sign, and -vmm_gen_ca; and a *development* (`DEV`) machine which has a fully featured OCaml -and MirageOS environment. Each step is prefixed with the machine it is supposed -to be executed on. Of course you can conflate everything into a single -development system or your server, all up to you and your security scenario. +Init scripts for FreeBSD are provided in the `packaging/rc.d` subdirectory. -Exact file transfer operations between these machines is not in scope of this -document, but kept abstract as `COPY`. Some commands require superuser -privileges (use `sudo`, `su`, or `doas`), I prefixed them with `#`. +TODO: from here on, this documentation is not up to date. -File endings used in this document: -- `.db` for CA databases -- `.pem` for (PEM-encoded) signed certificates -- `.key` for (PEM-encoded) private keys -- `.req` for (PEM-encoded) certificate signing requests - -## Setup a certificate authority - -The first step is to setup a certificate authority (private key and CA -certificate). The CA private key can sign and revoke everything, you should -better keep it in a safe place (air-gapped machine etc.) - not on the server! - -``` -CA> vmm_gen_ca ca ca.db [--days 3650] [--server "server"] [--server-days 365] -``` - -This generated five files: -- `ca.key` which is the CA private key -- `cacert.pem` which is the CA certificate -- `ca.db` which contains a map between serial number and name of issued certificates -- `server.pem` is the server certificate -- `server.key` is the private key of the server - -## Server setup - -If you have installed this package on your development machine, follow some more -steps to produce the remaining required binaries: - -``` -CA> COPY cacert.pem server.pem server.key SRV: -DEV> git clone https://github.com/mirage/mirage-skeleton.git -DEV> cd mirage-skeleton/tutorial/hello -DEV> mirage configure -t hvt -DEV> mirage build -DEV> mv solo5-hvt /tmp/solo5-hvt.none -DEV> cd ../../device-usage/network -DEV> mirage configure -t hvt -DEV> mirage build -DEV> mv solo5-hvt /tmp/solo5-hvt.net -DEV> cd ../../.. -DEV> COPY /tmp/solo5-hvt.none /tmp/solo5-hvt.net SRV:/var/db/albatross -DEV> COPY vmm_console vmm_log vmm_stats_lwt vmmd SRV:/opt/bin/ -``` - -``` -SRV> vmm_console -vv & -SRV> vmm_log -vv & -SRV> vmm_stats_lwt -vv & #optional -SRV# vmmd -vv cacert.pem server.pem server.key -``` - -Some setup for network interfaces is needed, depending on your operating system. -You can also add NAT to allow your virtual machines to talk to the outside -world, or add your external interface to the bridge directly, or just keep your -VMs local. - -``` -# FreeBSD -SRV# ifconfig bridge create #should output bridge0 -SRV# ifconfig bridge0 name ext -SRV# sysctl net.link.tap.up_on_open=1 -# Linux -SRV# brctl addbr ext -``` - -At least on FreeBSD, in order to monitor unikernels write permissions to -`/dev/vmm/` are needed. To achieve this (otherwise `vmm_stats` won't be -able to collect statistics unless running as a privileged user, the following -`devfs` ruleset can be used in `/etc/devfs.rules` (in case you created an -`albatross` group): - -``` -[albatross=10] -add path 'vmm/solo5*' mode 0660 group albatross -``` - -Also need to activate by adding `devfs_system_ruleset="albatross"` to -`/etc/rc.conf` and `service devd restart` on the host system. - -## Provision our first virtual machine - -We will delegate some resource to a certificate and key we keep on our -development machine. - -``` -DEV> vmm_req_delegation dev 2 1024 --cpu 1 --bridge ext/10.0.0.2/10.0.0.5/10.0.0.1/24 -DEV> COPY dev.req CA: -``` - -This produced two files, dev.req and dev.key. Keep the key in a safe place! - -``` -CA> vmm_sign ca.db cacert.pem ca.key dev.req [--days 10] -CA> COPY dev.pem DEV: -``` - -Now, our DEV machine can use its delegation certificate for issuing other -certificates. We'll create a certificate for interactive use, and one -containing the hello unikernel. - -``` -DEV> vmm_req_permissions admin --permission all -DEV> vmm_sign dev.db dev.pem dev.key admin.req -``` - -This produced in the first step two files, `admin.req` and `admin.key`, and in -the second step two more files, `dev.db` and `admin.pem`. - -``` -DEV> vmm_req_vm hello mirage-skeleton/tutorial/hello/hello.hvt 12 1 -DEV> vmm_sign dev.db dev.pem dev.key hello.req -``` - -This generates a private key `hello.key` and a certificate signing request named -`hello.req` including the virtual machine image `hello.hvt`, which gets 12MB -memory and CPU id 1. The second command used the `dev.key` to sign the signing -request and output a `hello.pem`. - -The flag `--force` can be passed to `vmm_req_vm`. This means: if there already -exists a running virtual machine with the same name, kill it and start the new -one provided in the certificate. - -To actually deploy anything, the server needs the chain (i.e. the vm certificate -and the delegation certificate). Our client needs the main CA certificate to -authenticate the server itself. - -``` -CA> COPY cacert.pem DEV: -DEV> cat admin.pem dev.pem > admin.bundle -DEV> cat hello.pem dev.pem > hello.bundle -``` - -And deploying (watch the output of the processes started on the server above!): - -``` -DEV> vmm_client cacert.pem hello.bundle hello.key SRV:1025 -DEV> vmm_client cacert.pem admin.bundle admin.key SRV:1025 --db dev.db -``` - -Commands are at the moment `info`, `statistics`, `destroy`, `attach`, `detach`, -and `log`. +It may help to read [the _outdated_ blog article](https://hannes.nqsb.io/Posts/VMM) +for motivation of albatross and an overview over its functionality.