reynir/reynir.dk
1
0
Fork 0
Code Issues 5 Pull requests Packages Projects Releases Wiki Activity

Import posts

Browse source 
May need some fixing
This commit is contained in:
Reynir Björnsson 2023-11-14 09:27:39 +01:00
parent 29e249d854
commit 86987925f3
8 changed files with 600 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

30
posts/2013-10-19-12-00-hakyll.markdown Normal file
View file

@ -0,0 +1,30 @@
---
title: Hakyll
date: 2013-10-19
---
I decided to check out [Hakyll](http://jaspervdj.be/hakyll). It is a static
site generator written in [Haskell](http://haskell.org). It seems to be very
flexible. It allows me to write the content in a boat load of different formats
(`.html`, `.rst`, `.markdown`, …).
## Static sites
I don't feel I have a need for dynamic content at the moment. And if I do need
dynamic content, I can make a *web app* for it and configure *nginx*
appropriately. For example, if you click on the picture of [Mario](/) you should
hear a sound. This uses websockets - all visitors should hear the same sound at
(approximately) the same time.
I find static site generators appealing. They can help with the tedious parts of
creating a web site - Creating indices, templating, balancing start- and end
tags, and so on.
## Haskell
Haskell is a *pure, lazily evaluated, functional programming language*. I have
some prior experience with Haskell, but I have never been a big fan of the
language. I think it's partly because I never really started using Haskell for
anything but toy programs.
I like Hakyll so far.

21
posts/2013-10-19-14-35-intertubes.markdown Normal file
View file

@ -0,0 +1,21 @@
---
title: Intertubes
date: 2013-10-19
---
![Mario](/images/mario.jpg)\
You may have noticed the picture of
[Mario](http://en.wikipedia.org/wiki/Mario_(character)) on the [frontpage](/).
The picture was taken a few years ago. It was the door to the office I shared
with a few fellow students.
You may also have noticed what happens when you click on the picture. A sound is
played - a man saying "it's- it's- it's a series of tubes!" If you've been
visiting the site from multiple devices you should have heard the sound from all
of the devices!
One day I was thinking: "Hm, Mario is a plumber. A plumber fixes the tubes. The
internet is a series of tubes!"
[Intertubes](http://en.wiktionary.org/wiki/Intertubes)

10
posts/2014-10-02-12-25-Angular.js.markdown Normal file
View file

@ -0,0 +1,10 @@
----
title: AngularJS training
date: 2014-10-02
----
Yesterday I attended a great [AngularJS](https://angularjs.org/) training session by [Matias Niemelä](http://www.yearofmoo.com/) after having volunteered for the [goto Aarhus](http://gotocon.com/aarhus-2014/) conference.
I had only had a little exposure to AngularJS beforehand.
My expectations were somewhat low to be honest - not that the description made it sound bad, but my interests are more about programming languages.
I was pleasantly surprised.

28
posts/2014-10-05-17-45-DigitalOcean tcpdump?.markdown Normal file
View file

@ -0,0 +1,28 @@
----
title: DigitalOcean tcpdump?
date: 2014-10-05
----
Lately I have been working on implementing v2 of DigitalOcean's API.
While doing that I ran a query to get all actions on my account.
I discovered that the result by default is *paginated* with 25 results on each *page*.
I re-ran the query but asking for up to 1000 results.
While inspecting the result to see if I did get all 87 results this time I discovered this odd-looking action:
{
"id":XXX,
"status":"completed",
"type":"tcpdump",
"started_at":"2014-04-28T08:26:51Z",
"completed_at":"2014-04-28T08:28:00Z",
"resource_id":YYY,
"resource_type":"droplet",
"region":"ams1"
}
The *type* field has the value *tcpdump*.
To my knowledge I have never initiated a tcpdump on any of my droplets, and I didn't think it was possible.
In fact, I haven't been able to verify that it is or has been possible at all!
I'm still happy with DigitalOcean, although this looks a bit suspicious - I have no idea why they would run a tcpdump on my droplet.

156
posts/2015-11-08-14-10-SSH certificates with Gitolite.markdown Normal file
View file

@ -0,0 +1,156 @@
---
title: SSH Certificates with Gitolite
date: 2015-11-08
---
[Gitolite] is a "an access control layer on top of Git, providing fine access control to Git repositories."
It allows you to host several repositories for several Gitolite users using a single unix user.
A typical setup uses ssh for authentication.
A new user is added by commiting a public key in `keydir/` to the gitolite-admin repository.
The gitolite username corresponding to the public key is determined by the filename of the public key commited to gitolite-admin.
For example, `keydir/host1.example.com/puppet-reader.pub` corresponds to the gitolite user `puppet-reader`.
The directory `host1.example.com` doesn't carry any semantics, and is useful for when a user has more than one SSH key.
This post is about **how to add new ssh keys for a user *without* having to push the new public key to the gitolite-admin repository *every* time**.
This can be convenient when automatically generating keys for reading repositories containing e.g. code or configuration.
An example [script](#script) is embedded at the end of the post.
First we need to know a little about how Gitolite works with SSH authentication.
## How does this work?
When a new key in `keydir/` is pushed to the gitolite-admin repository a git hook script is run.
This script will read the public key and add it to authorized\_keys for the gitolite unix user.
The entry will look something like the following:
```
command="/path/to/gitolite-shell rbj",no-port-forwarding,[...] \
ssh-rsa AAAAB3N[...] \
rbj@laptop
```
That is, it is like a regular authorized\_keys entry, but with a certain set of options:
- The first option, `command="/path/to/gitolite-shell rbj"`, is what makes gitolite work with ssh.
What this option does is whenever I log in as the gitolite user using the corresponding private key the gitolite-shell command will be run instead of the original command or a login shell.
More on this later.
- The other options are all of the form `no-OPTION`.
These options further restrict what you can do with the ssh key,
for example it disallows the user from forwarding ports or transfering files with sftp.
### So what does gitolite-shell do?
Well, it does a bunch.
It is similar to git-shell, but using gitolite's access control.
But basically it checks the user supplied as first argument and `$SSH_ORIGINAL_COMMAND` against the permissions configured in `conf/gitolite.conf`.
If the user is allowed then the git commands are performed.
## Using certificates
This works great.
However, it can be a bit inconvenient to add new keys, especially if you do it repeatedly or want to automate it.
This is where [OpenSSH certificates] come to our rescue.
![Searching for 'ssh certificates' is notoriously difficult.](/images/did-you-mean-95f89861f324e4d9768d2a7393f6f8e2.png)
An OpenSSH certificate is a signed file containing a public key and some more information.
The certificate is signed by a good old ssh key known as the certificate authority (CA).
When creating and signing a certificate we can specify what user(s) the certificate is valid for, and a number of ssh options.
I recommend reading the [CERTIFICATES] section of the ssh-keygen man page for more information.
### The certificate options
- Passing `-n git` only allows the certificate to be used for logging in as the `git` unix user.
- The `-O clear` option unsets all options, and is equivalent to a series of `no-FOO` in authorized\_keys.
- The `-O force-command` option does exactly the same as `command="[...]"` in authorized\_keys.
An example
``` bash
ssh-keygen -s /path/to/CA -I test -n git \
-O clear -O force-command="/remote/path/to/gitolte-shell rbj" \
id_rsa.pub
```
### Trusting the CA
To trust a CA we simply add the key to authorized\_keys with the `cert-authority`.
For example,
```
cert-authority,no-port-forwarding,[...] ssh-rsa AAAAB3N[...] rbj@laptop
```
The `no-FOO` options are redundant if you passed `-O clear` when signing the key, but is nice to have just in case.
### Security caveats
Anyone with the CA key can sign keys that are valid for the git unix user.
They cannot get a PTY or start an sftp session, but they will still be able to run any command(!)
They can also sign a certificate valid for any gitolite user, of course.
Reading the man page this sentence stood out to me:
*"If both certificate restrictions and key options are present, the most restrictive union of the two is applied."*
I figured I could then restrict the cert-authority to one command (one gitolite user) like this:
```
cert-authority,command="/path/to/gitolite-shell rbj",no-port-forwarding,... \
ssh-rsa AAAAB3N... \
rbj@laptop
```
Unfortunately, OpenSSH seems to prefer the certificate's `force-command` option over the `command` option from authorized\_keys!
**update:** After writing this post I found the following in the sshd(8) man page. *Also note that this command may be superseded by either a sshd\_config(5) ForceCommand directive or a command embedded in a certificate.*
#### Conclusion
You have to trust the CA to only sign keys for the gitolite-shell command and for the gitolite users you want.
# Sample script
``` {#script .bash}
#!/bin/sh
# Modify these constants to your needs
cakey=/root/puppet-ca/puppet_ca
# This file contains a single number as a string
serial_file=/root/puppet-ca/serial
# You probably want to change this
gitolite_user=puppet-reader
# Update this to the correct path to gitolite-shell
# Check ~git/.ssh/authorized_keys if in doubt
gitolite_shell=/srv/git/gitolite/src/gitolite-shell
usage() {
echo Usage:
echo "$0 PUBKEY KEYID"
echo
echo PUBKEY is the key to be signed
echo KEYID should be a unique name
}
if [ "$#" -ne 2 ]; then
usage
exit 1
fi
pubkey=$1
keyid=$2
# Not the most robust code
serial=$(cat "${serial_file}")
echo $((serial+1)) > "${serial_file}"
# You might want to pass a validity period, e.g. -V +52w
ssh-keygen -s "${cakey}" -I "${keyid}" \
-n 'git' -O clear \
-O force-command="${gitolite_shell} ${gitolite_user}" \
-z "${serial}" "${pubkey}"
```
[1]: https://en.wikipedia.org/wiki/Git_(software)#Git_server
[Gitolite]: http://gitolite.com/
[OpenSSH certificates]: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-keygen.1?query=ssh-keygen&sec=1#x434552544946494341544553
[CERTIFICATES]: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-keygen.1?query=ssh-keygen&sec=1#x434552544946494341544553

79
posts/2017-03-20-11-27-Marrakech 2017.markdown Normal file
View file

@ -0,0 +1,79 @@
---
title: Mirage hack retreat 2017
date: 2017-03-20
---
Two weeks ago I was in Marrakech to attend the third [Mirage](https://mirage.io/) [hack retreat](http://marrakech2017.mirage.io/).
It was the first time I was
* attending the hack retreat,
* in Morocco,
* and outside Europe (although just barely)!
After landing in the newly expanded airport I managed to meet up with a few other attendees.
Luckily for me one of them had been to the hostel-turned-artist-collective(?) [Priscilla Queen of the Medina](http://www.queen-of-the-medina.com/en/index.html).
Otherwise I would have to [find my own way](https://www.youtube.com/watch?v=zgzwmyxlKBE)!
After about three walks I learned the route myself.
![RAK airport (* TODO: scale image *)](/images/RAK.jpg)
In Marrakech the weather was really hot, and the sun strong.
A week prior to arriving, I had had some pre-heating with 17 degrees in Vienna.
It was still quite a big jump in temperatures;
some days it was around 30 degrees, and with sun as strong as it gets on the warmest summer days in Denmark.
The situation was made worse by me somehow forgetting to pack my shorts (I swear I put them out on the desk at home so as to not forget them!)
# When not relaxing
When I was not relaxing, I spent most of my time working with [Michele](https://github.com/mmaker) and others updating his [Let's Encrypt / ACME client](https://github.com/mmaker/ocaml-letsencrypt).
Before arriving I had mentioned I had mentioned on the mailing list that the Let's Encrypt client was something I wanted to work on.
My motives were, however, mostly about getting something I could start using rather an interest in working on this.
It turned out to be a lot of fun to work with Michele and others on this.
A lot of my time on this project went in to handling accepting the terms of service.
To this day it's still a mystery how exactly you're supposed to do this.
The implementation works for most use cases, but there is still a corner case where it doesn't work:
A new client registration was initiated but not completed e.g. due to network errors - it's then a mystery to me how one is supposed to retrieve the terms of service.
Besides the ToS handling, we managed to implement some cool features!
## Getting distracted
While working on the Let's Encrypt client I had an idea and a sudden urge to implement this idea.
In utop you can `#show` the type of values and the structure of modules.
However, if you want to access the documentation strings you have to fire up another program called `ocp-browser` (from the package `ocp-index`; hit alt+h to access help information in `ocp-browser`).
This is a distraction to my work flow.
I thought to myself *'why is this not accessible from within utop?'*
So I started investigating how `ocp-browser` works, and I discovered `ocp-index` provides a library.
Perfect!
So I spent half a day trying to figure out how to get the path to `lib` inside your current opam switch using `opam-lib`.
After finally figuring this out I could query the doc strings of functions using the library.
Next step was implementing this as a OCaml toplevel directive.
Thanks to [Rudi](https://github.com/rgrinberg) I avoided spending another half-day trying to figure out how to do something not-so-well-documented.
It culminated in a repository which can be pinned and installed in opam as [ocp-index-top](https://github.com/reynir/ocp-index-top).
I ended up removing the `opam-lib` dependency and shelling out to `opam config var lib` instead as `ocp-browser` does.
I found this inelegant, but unfortunately the `opam-lib` API seems to be unstable and resulted in a couple of version conflicts and downgrades when other people wanted to install my package.
It's still a bit of a hack but *It Works For Me*™
## An SSH library
Another project I worked on (barely) was [awa-ssh](https://github.com/haesbaert/awa-ssh) by [Christiano](https://github.com/haesbaert).
It's a SSH library in pure OCaml.
It was a project I was excited about because SSH is a protocol I use often and many times a day, and I find the protocol itself interesting.
For example, the protocol can be used [as a chat client or for RPC](https://medium.com/@shazow/ssh-how-does-it-even-9e43586e4ffc).
Christiano suggested I first read (some of) the RFCs for the protocol in order to get a basic understanding.
I did so, and the next day I had read up some on the first steps in the protocol.
He then showed me how the code was structured, and I played around with a test server.
Immediately I found a bug!
My client sends a version string `SSH-2.0-OpenSSH_7.4p1 Debian-6`.
The bit before the space character is the version information while the bit after is a comment string.
The dash (`-`) in the comment threw off the parser, and it deemed it an invalid version string.
A test case was added and I then proceeded to fix the bug and got it [merged](https://github.com/haesbaert/awa-ssh/pull/2).
What a start!
I hope to spend some more time looking, and hopefully contributing, to this library.
…
Once I find time!
Overall it was a pleasant experience, and I look forward to joining next year!

129
posts/banawa-chat.md Normal file
View file

@ -0,0 +1,129 @@
---
title: MirageOS retreat & Banawá chat
date: 2023-05-07
---
In beginning of May 2023 I had the joy to participate in the 12th [MirageOS hack retreat][retreat] in Marrakech, Morocco.
There I met faces I know from previous retreats as well as many new faces.
Some I had interacted with online.
In either case it was good to see them.
At the hack retreat we discuss ideas, work on projects related to [MirageOS][mirage] and OCaml in general while also enjoying *very* nice food and warm weather.
As a result knowledge is exchanged, new ideas emerge and seeds for new exciting projects are made.
One project was to implement the server part of git by Jules and Paul-Elliot.
As part of this project (I believe) they wrote a small unikernel that acts as a ssh server.
Since I have some experience with [awa-ssh] from the [first MirageOS hack retreat][retreat-2017] I attended I tried to help them out.
The library is in an experimental state, and while the client part sees some use in the Mirage ecosystem the server part has largely been neglected for some time.
While discussing awa-ssh with them we talked about how authentication works in the library.
On every connection you spawn a server that has a list of users and their credentials (password, if applicable, and all their publick keys).
This interface is not ideal since on every connection you need to figure out all valid users and their credentials before the client even has suggested who they pretend to be.
This is something I (still) would like to change.
While on the subject I was reminded of ssh servers I had read about and tested where any public key is accepted and stored for future logins if the user doesn't already exist - a [Trust On First Use (TOFU)][TOFU] scheme.
I then got motivated to do some quick modifications to awa-ssh to allow this scheme.
## Try it yourself
Before I go too deep into the story I will first share the prototype online.
It is online at chat.reyn.ir.
It can be accessed using an ssh client:
ssh -i path/to/key_ed25519 myusername@chat.reyn.ir
Note that only ed25519 and RSA keys work at the moment.
The connection will silently hang if e.g. an ecdsa key is used (a fix is in the works).
Once connected you will be greeted by your username, and the last many messages are displayed.
At any time you can disconnect by pressing enter and then `~.` - this is the default escape sequence in OpenSSH to hard disconnect a session.
You can then reconnect using the same key, but trying to login as your username using a different key will not work.
That is, until the server is restarted.
There are many bugs, and the experience is not quite polished.
Below is the public key of the server and its fingerprint:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBEQRUNxL+cGcOOIGF0SXFqv0jbhXZJYcGhl6BctMtxN
SHA256:6nWSkqXCM/mBS/CoDzPkQLnHDLDYv/NqNXCOKfATKEE
## Back to the story
As mentioned the authentication flow is very opinionated and requires you to list all users before the ssh protocol begins.
This does not lend very well to the TOFU scheme I wanted.
As a result I replaced the list of users with a mutable hash table mapping usernames to public keys.
Then the authentication flow checks first if the user exists (a public key is associated with the username in the hashtable); if so, we verify the signature against that key.
Otherwise, we verify the signature against the public key provided by the client (just for good measure) and then add to the hashtable the username and the public key.
Now that key is trusted for that particular user for subsequent authentications - the Trust On First Use scheme.
It is possible to reserve usernames by associating them with some (bogus) public key, but otherwise you don't have any control over what usernames can be registered.
As this is not a nice property, and the stateful, mutable user database is quite a divergence from the original code I decided to rename my fork of awa-ssh.
Awa-ssh is named after the critically endangered indigenous people of Brazil, [Awá][awa].
Wikipedia has a category of indigenous peoples in Brazil, and there I learned of the [Banawá][banawa].
The fork was then renamed **banawa-ssh**.
This is all well and cool, but it just authenticating for no purpose is not so exciting.
I then started working on a chat service.
At the retreat we often have unreliable internet connection, and every now and then we use up all the prepaid data on our shared LTE connection.
For this reason we run an opam repository mirroring the official one (from the git repository) using our MirageOS implementation [opam-mirror],
and the [bob] file transfer tool.
A service I was missing was a messaging service for the local network.
My idea was this could be used to announce things such as:
> hey, I am going to the fruit market tomorrow morning - who wants to join?
or,
> is the internet down for everyone or is it just me? Should I go out and buy more internet credit?
Another point is we don't want to impose too much administration by requiring manual registration prior.
On the other hand it is preferrable to have some sort of nick names to distinguish senders, but we also don't people impersonating each other.
For this the TOFU scheme is a pretty good balance for a service on a local network.
The chat service is named Banawá chat, and it's source is available online on [github.com/reynir/banawa-chat](https://github.com/reynir/banawa-chat).
## First iteration of the chat service
The first iteration multiplexed messages from clients with their usernames to all other connected peers.
This worked okay as a proof of concept, but it had many pain points.
First, the server did not understand terminals.
This meant that each character entered was sent as separate messages(!)
Therefore the preferred way to use it was either by disabling pseudo-terminal allocation in the client (`ssh -T` in OpenSSH),
or by running a command (`ssh hostname lets-chat`).
Second, there was no back log upon connection, and users would need to stay connected at all times to ensure they don't miss out on any messages.
All in all a pretty miserable experience even if a working proof of concept.
## Second iteration
While talking with [@dinosaure][@dinosaure] about these issues he reminded me of his work-in-progress IRC client [catty] (or Ca-tty).
For that project he implemented a terminal interface layer for awa-ssh and MirageOS.
I then started copying much of his terminal user interface code with his help.
Quickly I realized that keeping separate message buffers for each connection or using and broadcasting new messages was way more complicated than the more obvious implementation of keeping a shared message backlog and rendering them in each client on updates using [Lwd][lwd] for tracking updates.
This both made the code simpler, and gave us the much desired message backlog.
Now the service was becoming useful.
During the development several bugs were discovered and many of those fixed and upstreamed to awa-ssh.
See for example:
- https://github.com/mirage/awa-ssh/pull/55
- https://github.com/mirage/awa-ssh/pull/56
- https://github.com/mirage/awa-ssh/issues/57
- https://github.com/mirage/awa-ssh/pull/58
The experience of working on banawa-chat has motivated me to work more on awa-ssh.
Ssh is a nice protocol with built-in authentication and encryption.
A ssh server could be embedded in many unikernels to provide access to inspect, control or modify another service securely.
I would like to give credit to Andrey Petrov who wrote a nice [blog post][ssh-blog] on things you can do with the ssh protocol.
He wrote as well a ssh chat server called [ssh-chat] that is available on `ssh ssh.chat`.
While it's been many years since I read his blog post and I had to dig around to find it it's been in the back of my head at the retreat.
[retreat]: https://retreat.mirage.io/
[mirage]: https://mirage.io/
[awa-ssh]: https://github.com/mirage/awa-ssh
[retreat-2017]: https://reynir.dk/posts/2017-03-20-11-27-Marrakech%202017.html#an-ssh-library
[TOFU]: https://en.wikipedia.org/wiki/Trust_on_first_use
[awa]: https://en.wikipedia.org/wiki/Aw%C3%A1-Guaj%C3%A1_people
[banawa]: https://en.wikipedia.org/wiki/Banaw%C3%A1
[opam-mirror]: https://git.robur.io/robur/opam-mirror
[bob]: https://bob.osau.re/
[@dinosaure]: https://blog.osau.re/
[catty]: https://github.com/roburio/catty
[lwd]: https://github.com/let-def/lwd
[ssh-blog]: https://shazow.net/posts/ssh-how-does-it-even/
[ssh-chat]: https://github.com/shazow/ssh-chat

147
posts/creating-of-builder-web.md Normal file
View file

@ -0,0 +1,147 @@
---
title: Announcing builder-web
date: 2022-03-08
---
# Lessons from writing builder-web
These days we in [Robur](https://robur.coop/) are finishing up our [NGI Pointer](https://pointer.ngi.eu/) project with a goal of easing deployment of reproducible [MirageOS](https://mirage.io/) applications.
Two components we built for this purpose are [builder](https://github.com/roburio/builder) and [builder-web](https://git.robur.io/robur/builder-web/).
In this post I will go into technical details, design decisions and how the code for builder and builder-web evolved.
My hope is the reader may gain insight into lessons we learned and how builder-web works.
First, let's get an overview of what the software does.
## Builder
Builder is a server that periodically schedules build jobs to connecting builder worker clients over TCP.
The workers then execute the scripts received, typically an invocation of [orb](https://github.com/roburio/orb) which builds an opam package and records information necessary to [reproduce](https://reproducible-builds.org/) the build.
The artifacts and exit code are then packaged and sent back to the builder-server.
Builder-server can then store the results in an ASN.1 encoded file on the filesystem, or if configured upload it to builder-web.
If the upload fails, the results are stored in the filesystem.
## Builder-web
We started writing builder-web in late 2020.
Builder-web's main purpose is to receive results from builder-server(s) and present the build results and artifacts to users in a web interface.
One can view the jobs, their builds and their results, download binary artifacts with checksums or retrieve the information necessary to rebuild and reproduce the artifacts.
It has been extended with comparisons of dependencies between builds, visualizations of opam dependencies and binary size, and a recent hook mechanism that we now use to publish built binary packages for FreeBSD and Debian to our package repositories ([apt.robur.coop](https://apt.robur.coop/), [pkg.robur.coop](https://pkg.robur.coop) for debian and FreeBSD respectively).
We run builder-web at [builds.robur.coop](https://builds.robur.coop/) since end of 2020.
It has been running almost continuously since.
This is thanks to our effort in writing migration (and rollback) "scripts" at every time we needed to make a database schema change or alter the database in other ways.
I write "scripts" because they are OCaml modules that get built into the binary `builder-migrations`.
If you are curious about the history of database modifications in builder-web and have builder-web built or installed locally you can run `builder-migrations --help` or just `builder-migrations` and a man page is printed out with a list of sub commands and their descriptions.
![Help page of builder-migrations showing fixup and migration subcommands](builder-migrations.png)
Another contributing factor is that we use ZFS on the filesystem.
Before running a migration or operation that alters the files stored in the filesystem outside the database we can make a ZFS snapshot.
If the operation say deletes the wrong files we can do a quick rollback to the ZFS snapshot.
Running software that is actively being developed on is bound to have things go wrong, and we have had things go wrong a number of times.
But thanks to the (sometimes redundant) precautions we have always been able to quickly roll back and continue.
### Evolution of builder-web
The first iteration of builder-web was a very simple opium application that looked into the directory where builder-server would save the results, and would parse the the ASN.1 encoded files and present it in a web interface.
This allowed anyone to view the builds and download the artifacts but not much else.
Not long after the application was redesigned to have an upload endpoint where builder-server could upload build results which are then parsed and stored in a sqlite3 database and on the filesystem.
This made the two applications more independent and builder-web would no longer need access to the same filesystem as builder-server uses.
We chose to use sqlite3 through [caqti](https://github.com/paurkedal/ocaml-caqti) because an embedded database like sqlite3 means no setup of a database server is required, and sqlite3 has good performance.
Our hope in choosing caqti was that we could easily switch to a database server if the need should arise, but it turns out the differences between sqlite3 and e.g. PostgreSQL are large enough that many SQL queries will have to be rewritten.
So far we have not had a need to switch to using a database server, but I believe it should not be too big a hurdle to overcome.
In the web app itself the functions doing the queries are concentrated inside [`lib/model.ml`](https://git.robur.io/robur/builder-web/src/branch/main/lib/model.ml).
Another lesson is that storing data in files on the filesystem tracked by the database can greatly reduce the size of the database.
Even for small files such as short shell scripts.
#### Using phantom types to catch using IDs for the wrong table
When working with databases a good practice to use unique IDs in tables.
Foreign keys in other tables can then refer to a row in that table by its ID.
This ID is often a 64 bit integer.
A problem is that on the OCaml side you get a 64 bit integer and the programmer has to carefully track what table that ID refers to.
This is both annoying and error prone.
A solution we came up with is a custom Caqti type `'a id`.
```OCaml
module Rep = struct
type 'a id = int64
let id (_ : 'a) : 'a id Caqti_type.t = Caqti_type.int64
end : sig
type 'a id
val id : 'a -> 'a id Caqti_type.t
end
```
This allows us to write queries with typed IDs:
```OCaml
struct
let get_id = Caqti_request.find Caqti_type.string (id `my_table) "SELECT ..."
...
end : sig
val get_id : (string, [`my_table] id, _) Caqti_request.t
val use_id : ([`my_table] id, int, _) Caqti_request.t
val use_id_for_other_table : ([`other_table] id, int, _) Caqti_request.t
end
```
With the above queries it is no longer possible to erroneously use the ID retrieved using the `get_id` query with the query `use_id_for_other_table` without the type checker complaining.
In June 2021 we migrated from [opium](https://github.com/rgrinberg/opium) to [dream 1.0.0~alpha2](https://github.com/aantron/dream).
If I remember correctly one of the motivations was that we were not satisfied with how opium's opinionated logging works.
The migration went surprisingly smooth.
The main changes we had to make were few:
The functions in opium for creating a response return the response while in dream they return them in a lwt task.
Another difference is that dream does not have built-in support for tyxml unlike opium so we had to write a function `string_of_html` and change from `... |> Response.of_html` to `... |> string_of_html |> Dream.html`, roughly.
Dream also comes with middleware for built-in caqti lwt pools which is really convenient.
#### Multiple build platforms
On [builds.robur.coop](https://builds.robur.coop/) we build packages for various distributions.
We started building only on FreeBSD but soon added ubuntu, debian 10 and 11, mainly building system packages such as albatross, solo5-hvt tenders etc.
One reason for adding more platforms was of course to have packages for more platforms, but also to exercise the builder code on more platforms.
Initially, we set up a pair of builder-server and builder-worker for each new platform and scheduled jobs named e.g. albatross-debian-10.
This proved inflexible and suboptimal.
That architecture did not allow one builder-server to schedule jobs for different platforms.
A somewhat extensive redesign was initiated.
Every part needs to be aware of what platform builds are performed on.
Builder-worker was extended to take an extra command line parameter with the platform string which is passed on to builder-server when asking for build jobs.
Builder-server needs platform-specific orb templates.
Interesting questions arise such as what does it mean for scheduling if a client orders an execution of a job on only one platform, and what should happen when a builder-worker connects with a new platform string or an orb template for a new template appears or disappears in the file system.
The ASN.1 format also needs to convey what platform the build was built on.
A new usability challenge also occurred:
The build platform is important for system packages, but for hvt unikernels the build platform matters less; it is of interest if the user wants to reproduce the build on their own, but for a user that is only interested in downloading and running the unikernel the build platform is likely of very little interest.
Our solution is to keep the build platform listed and put a note on the main page that for unikernels the execution target is independent of the build platform.
Other solutions and suggestions are very welcome either as an [issue on Github](https://github.com/roburio/builder-web/issues) or by email: team ATrobur.coop.
#### Upload hooks
We added some in my opinion very nice and useful analyses and visualizations that my coworker wrote more about on [his blog](https://r7p5.earth/blog/2022-3-7/Builder-web%20visualizations%20at%20Robur).
One of the analyses is based on parsing ELF symbol tables, and it proved to be too slow for a web server, roughly 1-2 seconds.
To solve this problem we need to cache the visualizations.
Generating the visualizations on first hit would give a poor user experience where the user may experience very slow load times if they happen to be the first to request the visualization.
Instead we asynchronously execute a binary on upload that generates the visualizations.
We were also interested in creating and updating system package repositories with binary packages.
Therefore we decided to define an upload hook script interface.
Operators can write scripts and put them in `/etc/builder-web/upload-hooks/` (or `/usr/local/etc/builder-web/upload-hooks` on FreeBSD), and then the scripts are executed on upload with information about the build passed as command line arguments.
We use this for generating the two analysis visualizations and for updating binary system package repositories for FreeBSD and debian family distrubutions.
With some minor breaking changes the mechanism could be extended so that it can be used for sending reports about failed builds.
The debian repository is available at [apt.robur.coop](https://apt.robur.coop/) while the FreeBSD repository is available at [pkg.robur.coop](https://pkg.robur.coop).
#### Unikernel device manifest
While we were working on ELF symbol table analysis I came upon [owee](https://github.com/let-def/owee) as a dependency of another dependency.
It is an OCaml library for parsing ELF files and more.
For a while I had been working on a parser for a subset of the [ELF format](https://en.wikipedia.org/wiki/Executable_and_Linkable_Format) in order to retrieve solo5 device manifests from Mirage unikernels.
With the tedious ELF parsing already implemented I wrote [ocaml-solo5-elftool](https://github.com/roburio/ocaml-solo5-elftool) to be used in [albatross](https://github.com/roburio/albatross).
In albatross we used the solo5-elftool binary from [solo5] to query unikernel images for what devices they expect in order to fail early if the user did not specify exactly those.
This sort of error can be annoying so I added this functionality to builder-web to display the required devices for each unikernel.
## Closing notes
A lot was learned in writing builder, builder-web and the other projects we worked on for the EU NGI Pointer project.
I believe the work we have done the past year is a good step towards a good story for deploying and running Mirage applications.
I would like to thank NGI Pointer for the generous funding and support that made this work possible.