“Baby steps.”
I’ve added a single change today. There’s new role that will install and configure the DigitalOcean monitoring agent.
In the coming days, I’ve got a few more items to wrap up. Once those (minor) missing pieces are in place, I will call this project done and move on to something else, like re-re-architecting this website using Kubernetes.
You can see what’s new here: https://github.com/seaburr/WordPressOnDigitalOcean
Have a good weekend.
I can’t believe it’s been over two weeks since I touched this project. So far, things have been good. I took a look at AWS with the intent of redoing this project on Lightsail. I assumed it would be cheaper. Unfortunately, it won’t be. That doesn’t mean it’s not worth the effort, but it certainly changes the time table to migrate because it will actually cost MORE per month than hosting on DigitalOcean. Instead of focusing on migrating, I’ve decided to focus on general work that needs to be done which translates to any other cloud vendor.
Here’s what’s new:
You can see what’s new here: https://github.com/seaburr/WordPressOnDigitalOcean
Have a good weekend.
“Bootstrapped.”
Last night, I migrated this website onto its new infrastructure! The tooling I’ve been working for the last week has become feature-complete enough that it was used to create the database, droplet, and storage volume this site now uses.
There was some manual work (SSL configuration, import from previous website, updating DB connection string) however it was relatively minor and these things are likely to be the next features added.
Here’s where we’re at:
I’m really excited about the progress made in a week.
“Biting off more than I can chew.”
As of Sunday, I had gotten droplet creation, volume storage, and a good bit of the base OS setup work handled. The major missing piece was the creating the database. I am happy to say that work is now complete, but it’s not without bumps, warts, and bruises.
Creating the database server proved daunting because Ansible modules do not exist for DigitalOcean’s managed database server. I thought I’d just write a module or two, however that proved to be a much more daunting task than I first realized.
I’ve never written a module for Ansible, I’m no DBA, and the number of APIs exposed for database actions left me feeling deflated. Still, I soldiered through and created a Python script I can use to create, configure, or destroy a managed MySQL server. There are a ton of assumptions made to keep the scope of work reduced, but it’s sufficient for my needs at this point. It’s wrapped with argparse so I can run it from an Ansible command task then parse the JSON output.
Check it out here: https://github.com/seaburr/WordPressOnDigitalOcean/blob/master/roles/database-server/files/digital_ocean_database.py
As I said, I believe Ansible modules should exist for their managed database, as they do for AWS and GCP however, I think that’s a larger project than one individual can tackle on their first attempt writing modules. I’d be happy to collaborate on that if it ever comes up.
In related news, I’ve also extended the install-wordpress role to create a wp-config.php from template, including fetching unique salts from wordpress.org, I’ve also fixed a few small issues with the Apache installation and added missing packages.
At this point, the only thing missing to go from nothing to a functioning WordPress site is adding database connection information into wp-config.php and handling SSL.
“Automate all the things!”
This weekend I’ve been busy laying much of the groundwork requires to be able to re-deploy this website. In part one, I provided a diagram that showed how the entire website, including the database, is deployed on a single droplet (VM). While this model has worked well for about three years, it’s time to move on. To that end, I’ve begun building automation to create and manage WordPress sites.
I’ve created a repo on Github to house the infrastructure code. It’s available here: https://github.com/seaburr/WordPressOnDigitalOcean
As you can see, I’ve decided to continue using DigitalOcean. They’ve proven reliable. Unfortunately, DigitalOcean lacks some things, like NFS, that would make the website even more scalable. Still, I don’t feel they’re necessary at this point.
Currently, I have the ability to create/destroy VMs, create/destroy/attach storage volumes, and a decent portion of the application installation is complete. It’s capable of going from nothing to the WordPress setup page over HTTP.
Finally, I know I could just move all of this to AWS LightSail but what’s the fun in that?? I learn best by doing and I want to have a deep understanding with all the moving parts. Once I’m done building this out with DO, I might do the same on the other major cloud vendors.
We’re onto part two! Luckily, this is a simple change.
The VM that runs this website is on DigitalOcean. It’s so old (3 years) that it was only provisioned with 1CPU/512MB of memory. This has been sufficient but it’s a tight squeeze with the reverse proxy and MySQL shoehorned onto the same VM. Tonight when I ran OS updates, I noticed this website went down. After looking further, the Linux OOM killer had stepped in and killed MySQL to keep the machine up and running. After starting MySQL, I scaled the droplet to a larger size.
DigitalOcean allows you to scale your droplet but they cannot perform hot add, so the droplet needed to be powered off. Luckily, the $5/mo droplet size has increased to 1CPU/1GB so I was able to double this VMs memory at no cost.
Let’s call this our “get well plan” so the next time updates are run, this site doesn’t go down.
Let’s embark on a journey to re-architect this website to something that is more resilient. Currently, the entire website runs on a single DigitalOcean droplet.
Backups are handled through a Bash script that zips up the application directory, dumps the DB, and tars up the output.
This could be better. Let’s make this better. As the website gets re-architected, I’ll provide diagrams and links to the code & documentation leveraged.
It appears that the automated install of CentOS 8 does not work correctly in the latest version of VMware Workstation Pro 15.5 (15.5.0 build-14665864 at time of writing).
I’m not surprised considering it’s not even an option in the drop down but it’s good to know that there’s a workaround for this.
During an install , you will be prompted by the following error:
0016457: Pane is dead
The following problem occurred on line 31 of the kickstart file:
Section %packages does not end with a %endLuckily, the workaround is simple. When you create your VM, select “I will install the operating system later.” You can configure the hardware and boot the machine for the first time. Afterwards, attach a optical drive with the CentOS 8 ISO and reboot. You can then go through the regular installer process.
The issue is logged as a bug on CentOS Bug Tracker though I’m guessing this is actually an issue with Workstation.
https://bugs.centos.org/view.php?id=16457
Even more good news! CentOS 8 ships with a much, much more modern version of Git, 2.18.1 which released in September of 2018 (latest at time of writing is 2.23 which released mid August, 2019). You no longer have to download and build from source just to use HTTPS Git repos 🙂
I consider a role to be a reusable, (mostly) standalone component to be consumed within a playbook. For instance, we have several different tools which require JDK, so we have a single role called install-oracle-jdk which downloads the JDK, installs it, updates the cacerts with some of our internal certificates, and makes sure that Java is available in the path for all users.
In the most simple case, a role is broken down like this:
.
__roles/
|__role-name/
|__tasks/
|__main.yml
|__vars/
|__main.ymlIn side of the role-name directory are two directories, the tasks/ directory and the vars/ directory. Inside of those two directories is a file called main.yml. vars/main.yml contains the default values of the variables needed for a given role and tasks/main.yml contains the steps required to apply that role. The intent is that if a role is run within a playbook without any variables overridden (more on that later), it will use the default variables. For a role like install-oracle-jdk, there might be a need to specify a different version of JDK. The other benefit of this approach is that when we’ve gone from 8_181 to 8_191 to 8_201 (current at time of writing), we only need to change it in one place and the next time we run the playbook, we’re able to confirm that the same/latest version of Java is installed across all machines in the inventory file which consume that role.
We have other roles where we’d need to pipe in additional variables. When executing a playbook, we generally use the a defaults file which we keep up in the top level directory of our Ansible repository. In the next post, I’ll cover how/when/why we use these defaults files.
Over the next few weeks, I’ll be doing write-ups on how I upgrade some of our developer tools. Specifically, I’ll focus on the Atlassian applications I manage, but the process for our other tools (like Artifactory, SonarQube, etc.) is largely the same and most of the Ansible Roles are common across all of the applications.
For now, the playbooks and roles will not be available but I’ll post relevant snippets where I feel it’s needed.
Finally, we use RHEL/CentOS internally (for hosting our internal tools, we have other OSes for doing builds) so our playbooks are oriented towards a single OS.