From 2eb2c97a0855e850485af8549464cd7cc5b2ccb7 Mon Sep 17 00:00:00 2001 From: pierreozoux Date: Fri, 7 Nov 2014 15:37:47 +0000 Subject: [PATCH] Updates documentation --- README.md | 29 ++++++++++++++++++++++------- doc/architecture-overview.md | 27 +++++++++++++++++---------- doc/deploying-a-server.md | 2 +- scripts/activate-user.sh | 18 ------------------ 4 files changed, 40 insertions(+), 36 deletions(-) delete mode 100755 scripts/activate-user.sh diff --git a/README.md b/README.md index 506b829..626a425 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ ## IndieHosters -This repository contains the confd and bash scripts we use to control our servers. -It can run inside Vagrant (see below; FIXME: check whether these instruction currently work) or +This repository contains the configuration and scripts we use to control our servers. +It can run inside Vagrant or [deploy to a server](doc/getting-started-as-a-hoster.md) (FIXME: update those instructions to prescribe less folder structure, explain static https+smtp hosting, and check if they currently work). @@ -10,8 +10,8 @@ work). - [vagrant](http://www.vagrantup.com/) - [virtualbox](https://www.virtualbox.org/) - nfs - - run `apt-get install nfs-kernel-server`, or your OS equivalent -- optional: [vagrant-hostsupdater](https://github.com/cogitatio/vagrant-hostsupdater) + - linux: run `apt-get install nfs-kernel-server`, or your OS equivalent +- [vagrant-hostsupdater](https://github.com/cogitatio/vagrant-hostsupdater) - run `vagrant plugin install vagrant-hostsupdater` to install ## Get started: @@ -20,15 +20,19 @@ work). vagrant up ``` -Wait for the provisioning to finish (~5mins), and go to your browser: https://indiehosters.dev +Wait for the provisioning to finish (~12mins), and go to your browser: https://indiehosters.dev + If the process fails, for instance due to network problems, you can retry by running `vagrant provision`. -### If you want to add another static-git instance apart from indiehosters.dev: +### If you want to add another static instance apart from indiehosters.dev: ```bash vagrant ssh -sudo sh /data/indiehosters/scripts/activate-user.sh example.dev static-git https://github.com/indiehosters/website.git +sudo mkdir -p /data/import/example.dev/TLS +sudo cp /data/indiehosters/scripts/unsecure-certs/example.dev.pem /data/import/example.dev/TLS +sudo systemctl start static@example.dev ``` + Check https://example.dev in your bowser! ### Cleaning up @@ -39,3 +43,14 @@ To clean up stuff from previous runs of your VM, you can do: vagrant destroy vagrant up ``` + +## Tests + +```bash +vagrant destroy +vagrant up +# Check in your browser https://indiehosters.dev +vagrant ssh +sudo su +/data/indiehosters/scripts/tests/main.sh +``` diff --git a/doc/architecture-overview.md b/doc/architecture-overview.md index e220c31..21bf7e8 100644 --- a/doc/architecture-overview.md +++ b/doc/architecture-overview.md @@ -6,38 +6,45 @@ Our architecture revolves around a The haproxy.* and postfix.* unit files correspond to two server wide processes. They run Docker containers from images in the [server-wide/ folder of our dockerfiles repo](https://github.com/indiehosters/dockerfiles/tree/master/server-wide). -The haproxy-confd.* unit starts a side-kick service for haproxy, which monitors `etcdctl ls /services` to see if any new backends were created, and updates the haproxy configuration, which lives in `/data/runtime/haproxy/` on the host sytem. It is required by the haproxy.* unit. That means that when you run `systemctl start haproxy`, and then run `docker ps` or `systemctl list-units`, you will see that systemd not only started the haproxy container, but also the haproxy-confd container. +The haproxy-confd service unit starts configuration service for haproxy. It monitors `etcdctl ls /services` to see if any new backends were created, and updates the haproxy configuration files. The files lives in `/data/runtime/haproxy/` on the host sytem. It is required by the haproxy.* unit. That means that when you run `systemctl start haproxy`, and then run `docker ps` or `systemctl list-units`, you will see that systemd not only started the haproxy container, but also the haproxy-confd container. -There is currently no similar side-kick for updating `/data/runtime/postfix/`, so you will have to update the configuration files in that folder manually, and then run `systemctl restart postfix`. +There is currently no similar service for updating `/data/runtime/postfix/`, so you will have to update the configuration files in that folder manually, and then run `systemctl restart postfix`. -The `scripts/setup.sh` takes care of setting up etcd, enabling and starting the haproxy and postfix service (as well as one haproxy backend, to serve the default site), and the haproxy-confd side-kick to listen for changes in the backends configuration in etcd, so that new backends are automatically added to the haproxy config as soon as their IP address is written into etcd. +Etcd is configured in the `cloud-config` file. The `scripts/setup.sh` takes care of enabling and starting the haproxy and postfix service, and the haproxy-confd to listen for changes in the backends configuration in [etcd](https://github.com/coreos/etcd). New backends are automatically added to the haproxy configuration as soon as their private docker IP address is written into etcd. ## HAProxy backends: static, static-git, wordpress -A per user process is a haproxy backend for a specific domain name. At the time of writing we have three applications available: static, static-git and wordpress. +A per user process is a haproxy backend for a specific domain name. For now, we have three applications available: static, static-git and wordpress. But we are working on adding more. Please, if you want, you can create more :) -You will notice there are also some other units in the `unit-files/` folder of this repo, like the gitpuller and mysql ones. Whenever you start a wordpress unit, it requires a mysql service. -Whenever you start a static-git unit, it wants a static-git-puller unit. In all three cases, an -importer unit and a -discovery unit are required. +You will notice there are also some other units in the `unit-files/` folder of this repository, like the gitpuller and mysql ones. + +Whenever you start a wordpress unit, it requires a mysql service. + +Whenever you start a static-git unit, it wants a static-git-puller unit. + +In all three cases, an -importer unit and a -discovery unit are required. This works through a [`Requires=` directive](https://github.com/indiehosters/indiehosters/blob/0.1.0/unit-files/nginx@.service#L6-L7) which systemd interprets, so that if you start one service, its dependencies are also started (you can see that in `systemctl list-units`). ## Discovery -The -discovery units check find out the local IP address of the backend, checks if it is up by doing a `curl`, and if so, writes the IP address into etcd. The `haproxy-confd` process notices this, and update the haproxy config. +The `-discovery` units check find out the local IP address of the backend, checks if it is up by doing a `curl`, and if so, writes the IP address into etcd. The `haproxy-confd` service notices this, and update the haproxy config. ## Import -The -import units check if data exists, and if it doesn't, create initial data state, for instance by doing a git clone, untarring the php files of a virgin wordpress installation, or both. So -import is actually a misnomer, probably -init would have been a better name. +The `-import` units check if data exists, and if it doesn't, tries to import from the `/data/import` folder, and create initial data state, for instance by doing a git clone, untarring the content of a vanilla wp-content for wordpress installation. Note that some initialization is also done by the Docker images themselves - for instance the wordpress image runs a [shell script](https://github.com/pierreozoux/tutum-docker-wordpress-nosql/blob/master/run-wordpress.sh) at container startup, that creates the initial mysql database if it didn't exist yet. +We are working on merging everything inside the docker image. + ## Gitpuller -The -gitpuller unit is scheduled to run every 10 minutes by the .timer file, and is configure to only run if the GITURL file exists at the path specified in the .path file. When it runs, it does a git pull to update the website content at one of the haproxy backends from the git repository mentioned in the GITURL file. +The `-gitpuller` unit is scheduled to run every 10 minutes by the .timer file. When it runs, it does a git pull to update the website content at one of the haproxy backends from the git repository mentioned in the GITURL file. ## Scripts -There are two important scripts you can run at your server. You can also run the commands they contain manually, then you just use them as a cheatsheet of how to [set up a new server](https://github.com/indiehosters/indiehosters/tree/master/scripts/setup.sh) or [activate a new user](https://github.com/indiehosters/indiehosters/tree/master/scripts/activate-user.sh), respectively. +There is one important script you can run at your server. You can also run the commands they contain manually, then you just use them as a cheatsheet of how to [set up a new server](https://github.com/indiehosters/indiehosters/tree/master/scripts/setup.sh). There are also deploy scripts which do the same from a jump box, so you can orchestrate multiple servers from one central vantage points. They are in the [deploy/](https://github.com/indiehosters/indiehosters/tree/master/deploy) diff --git a/doc/deploying-a-server.md b/doc/deploying-a-server.md index 3b9e860..c7daf9e 100644 --- a/doc/deploying-a-server.md +++ b/doc/deploying-a-server.md @@ -37,7 +37,7 @@ Make sure you read [getting started](getting-started-as-a-hoster.md) first. * Unless you already have a TLS certificate for example.com, get one (from StartSSL or elsewhere), and concatenate the certificate and its unencrypted private key into one file. - * Make sure the TLS certificate is valid (use `scripts/check-cert.sh` for this), and scp it to `/data/import/example.com.pem` on k3. + * Make sure the TLS certificate is valid (use `scripts/check-cert.sh` for this), and scp it to `/data/import/example.com/TLS/example.com.pem` on k3. * Now ssh into k3, and if for instance 'wordpress' is the image you chose, run: systemctl enable wordpress@example.com diff --git a/scripts/activate-user.sh b/scripts/activate-user.sh deleted file mode 100755 index 3d48355..0000000 --- a/scripts/activate-user.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -eux - -if [ $# -ge 2 ]; then - DOMAIN=$1 - IMAGE=$2 -else - echo "Usage: sh /data/indiehosters/scripts/activate-user.sh domain image [gitrepo]" - exit 1 -fi -mkdir -p /data/domains/$DOMAIN/$IMAGE -if [ $# -ge 3 ]; then - GITREPO=$3 - echo $GITREPO > /data/domains/$DOMAIN/$IMAGE/GITURL -fi - -# Start service for new site (and create the user). This will also enable the git puller. -systemctl enable $IMAGE@$DOMAIN.service -systemctl start $IMAGE@$DOMAIN.service -- GitLab