You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Reynir Björnsson 236dec8dfd opam: dummy synopsis 3 months ago
app remarks from running 3 months ago
lib initial commit 3 months ago
LICENSE.md initial commit 3 months ago
README.md initial commit 3 months ago
builder.opam opam: dummy synopsis 3 months ago
dune-project initial commit 3 months ago

README.md

builder - scheduling and executing jobs

This consists of two programs, a client and a server. The single server contains a queue of jobs, which are consumed by a client. Any number of clients can be connected to the server.

The server keeps persistent state of the job queue (so restarts / crashes are dealt with). A client connects, provides some information about itself, and then waits for a job. Once a job is read and accepted, it is executed by the client. Resulting artifacts can be transferred by the client to the server. 2 The server has the ability to schedule jobs at regular intervals - similar to crontab.

Handled and unhandled error conditions:

  • client execution fails (timeout, restart, killed): not handled
  • client execution gets a signal: reported to server
  • client can’t write job data files -> failure is captured and reported
  • client can’t read job output -> logged to client’s console (without artifacts gathered)
  • client errors when submitting console output -> ignored
  • client errors when submitting build artifacts -> retry
  • there’s no explicit ACK for packets

Left to do:

  • client should queue up console output on server connection failure (and continue reading output at the same time)
  • client should inform when data passed to it and artifacts overlap (name / checksum)
  • client could sandbox the executed script a bit more (maybe?)
  • client should have a timeout for the script to be executed
  • separate stdout and stderr to be sent to the server?
  • UI
  • retrieve artifacts even if execution failed
  • the running jobs are not stored onto disk, which may result in unexpected behaviour
  • should instead once a job is scheduled the uuid and job information being dumped to disk already (also avoids dummy job dump)
  • directory traversals (server folds over output directory and collects files)