Optimising Personal Hosting

Over the last few years, I had a number of small side projects to either learn new technologies or experiment with new ideas. Some of these projects live and die within a year, but most of them actually stay for longer or require a quick deployment as I don’t want to be held back by my processes when I want to release a project quickly.

In August last year (2017) I’ve decided to optimise my hosting for these projects - automate everything and also reduce the costs, as with virtual servers and especially AWS the costs can escalate quickly.

This blog post is an overview of what’s changed, the current process and possible improvements in the future.

Previous method

Previously, I’ve used a separate server for every project as it’s more convenient in terms of deployments and managing dependencies. Everything was on the same server, and all dependencies are specific to a project.

This worked well in the beginning when you have just a few projects. The main benefits were the ability to separate dependencies, permissions, etc. But unfortunately, it had a number of drawbacks.

Costs

The first one is the cost, for every project you need a separate server or even set of servers. Which increases every time you want to build a new project. You can’t use the smallest instances, as most of the time they are running quite heavy dependencies. So, in reality, it’s $10 per project per month, sometimes up to $20.

Core server management

Each project needs to be updated on a regular basis, security patches, deprecated dependencies, the more moving parts you have, the more chances there are to break something.

Servers management

To manage all the servers I’ve used ansible and it works well. But since each server was a separate ansible playbook, and they were not updated on a regular basis, with lots of copy-pasting between projects. Also, once you update ansible version on your local machine and there are config changes, that’s it, you’d need to rewrite every playbook so it works next time.

Solution

After reviewing what I had, I’ve decided to move to a single dedicated server with no auto-scaling etc. Yes, it’s a bit more expensive upfront, but after I’ve summed up all the costs, it actually turns out much much cheaper.

Main benefits of a dedicated hosting were: more raw power per $, lots of RAM and lots of CPU cores, and having a single price.

Since many projects would actually use the same server it was important to separate the dependencies, and I’ve used Docker to package the projects, but leave main databases on the root server. Having separate images is also important as these projects might use completely different stacks, for example, NodeJS, Golang, Python, or PHP.

Automation

I’ve used ansible to automate the main server, it’s still useful as it allows to replicate the setup quickly and introduce the new changes and requirements.

Since I’m using many common dependencies between projects I’ve created a Docker base image which I extend in other projects, this again saves time.

The most important part is that I can build the project locally, test it, and upload to the Docker repository, and use a single command deploy it to the server.

There is a way to further automate the last step I’ve described, for example when the new tag in the git repo is added, it triggers the building, testing, and deployment process, so once it’s built and all tests pass it will release the code automatically.

Next steps

There are some bits which are still missing from the process, and they are on the schedule.

For example, automating backups, i.e. before each deployment, the DB and assets can be backed up.

Deploying projects with no downtime - there is a number of ways to do it, but you can run two docker images and simply switch the port in the web server config, this would require non-breaking data migrations though.

Additional tools on the server which allows you running data pipeline processes. There were a few requests recently which required running long running scripts on the server, and having a docker image with the code which is supposed to run these data pipeline sounds more bulletproof than just script in long-running session on the server directly.

So far

So far the main objective is achieved, it saved me money with my projects. In the process of migrating to the new hosting, I’ve actually stopped a few projects as I had no interest in developing them further.