May 2020

Ansible, k8s, Terraform and being frugal

Running GeeryDev is a labor of love. With its 14 monthly views, it makes for a minor hassle and a great playground for me.  And with that, it struck me one day, that I wonder if I could lower my $7 Heroku cloud hosting bill.  Don't get me wrong, Heroku really is a great value, and the free MongoDB addons treated me well.  My main concern was paying $7 for every terrible web app I wanna publish to cloud infrastructure just didn't seem good enough. Also note, I know some would argue that I can do this all on lambda to really reduce the bill (or on free GAE), and my answer is simply I just don't wanna do that.  I am interested in hosting my own databases and really enjoy my SSH access.


As someone who has used Kubernetes quite a bit in the past couple years, it was obviously my first choice for creating a multi-tenant architecture. I should have known better, but I didn't give much thought to the smallest potential cluster.  Not surprisingly, my attempts to create a single node cluster with any thing smaller than an n1-standard-2 (GCP machine type), and $48 wasn't gonna be great considering the budget.  I ran into insufficient CPU pod scheduling issues for just the default GKE pods that continually led to tainted clusters.  Overall, it makes it pretty clear that K8s is not a great investment until you really need to start being concerned with a resilient infrastructure.

Also worth noting, Docker restart policies are a nice default feature for ensuring an error prone process (or webserver) will start itself back up.  This made me feel a little better about giving up on k8s.


Terraform is the configuration tool I never knew how much I needed.  Building, tearing down, and repeating my Kubernetes CPU mistakes was a rather trivial experience using Terraform.  Creating an ec2 environment was an equally pleasant experience.  My first attempt at using a t3.nano instance ended with running out of memory before I could get the GeeryDev node container up and running, so I settled on a t3.micro. This puts me at about $4 a month.  For this size, I am also able to run my Image Color Analyzer on the same machine. You can see my setup here. Now, we were getting somewhere.


Giving up on Kubernetes, and choosing to provision my own multi-tenant infrastructure, meant I'd need to get my hands dirty a bit. Enter Ansible.  My deployments are rather simple, and I think this is the beauty of Ansible.  There is not a lot of configuration, and the simple use case is still rather simple.  For example, you can view the GeeryDev deploy here, and could probably understand most of it, without having seen the syntax before.  I would recommend Ansible for DevOps for those interested in learning more.  The ease of use for Ansible provides a real nifty workflow. The setup of the ec2 instance with docker, lets encrypt certbot, and starting containers left a trail of bash commands. It is rather trivial after first exploration to take those commands and move m into an ansible playbook.  Now, as soon as Terraform provides me with any desired infrastructure, I am just a few playbooks away from having GeeryDev up in another environment.  It also provides an amazing oportunity to finally get my Geerydev MongoDB backups scheduled.


Letting containers run is not without its problems in a resource constrained environment. The image sizes (need to be optimized, but) take up about a considerable amount of the disk space on the instance.  Also, letting logfiles grow without bounds is guaranteed to use the remaining space.  Introducing a simple logrotate config for all of the docker containers has proved immensely useful. # /etc/logrotate.conf /var/lib/docker/containers/*/*.log {     rotate 3     daily     compress     missingok     maxsize 20M     delaycompress     copytruncate }


So what's the final bill? Something like $4.50 a month for the ec2 instance and underlying EBS volume. I have much more flexibility to handle SSH access, and multi-tenancy.  However, we know this is only half the story.  I still need to maintain, monitor and publish more apps. So, I'm sure I'll be back to revisit these choices as my readership grows, but for now I'll call it a win.