DevOps

by cb on Cloud, DevOps, Random

Dead Air

It’s been several years since I last posted anything though it’s not because I haven’t been busy. My goal is to start writing up posts regularly but before I do, I feel I need to recap the last three and a half years.

I would recommend that everyone occasionally take time to document their accomplishments (professionally and otherwise).

The single most important change in my life is that I am now a father and my son is an absolutely incredible little kid. I hope I can continue to help him grow into a kind, caring, smart, driven person who can accomplish more than I could ever dream of in whatever realm brings him fulfillment.

Now, as for work…

In July of 2024, I left Broadcom (post acquisition of VMware) to join a startup in the Atlanta area. Between the 2021 and 2024 at VMware and the last several months at Airia, I’ve accomplished quite a lot.

  • Built a Prometheus exporter to run performance tests and detects issues within the VMware build environment. During one troubleshooting event, it was able to confirm that a NFS performance issue was solved nearly instantly whereas the previous test involved running an large build that took several hours.
  • Built a tool to migrate repositories between Artifactory instances including recreating virtual repositories and their relationships with the underlying local and remote repos. It was used successfully to migrate several terabytes of data to a new instance of Artifactory hosted in AWS.
  • Built a SCCM provisioner for GitHub as a gap-stop to onboard hundreds of users into GitHub while we built selected and tested an IdP solution.
  • Built countless tools/processes using GitHub’s REST and GraphQL APIs to assist with configuring GitHub as teams migrated from Perforce and Bitbucket.
  • Selected, tested, deployed, documented, and tuned Loki as the logging system of choice for troubleshooting issues and getting statistics out of log data.
  • Building out several of our systems for dealing with incidents including a customer-facing status page, deploying PagerDuty for alerts, and implementing an RCCA process using 5 Why’s.
  • Combed through every component of our system and ensured that we had telemetry and log data and either added a dashboard from the community or built a custom dashboard. I want to ensure we have observability into everything.

This list will continue to grow rapidly and there are several things I’d like to add however they are things I’ve only just begun to look into. In short, I am busy as hell.

 
by cb on Automation, DevOps, Process Improvement, SRE, Terraform

Using Terraform to Manage DigitalOcean Resources

I am a fan of DigitalOcean. What they lack in breath of services they more than make up for with the ease of use, documentation, and tutorials. Last year, I overhauled this website to be driven by Ansible. This year, I want to take this automation to the next level. There are capability gaps using Ansible to create infrastructure that I’ve had to work around by doing some tasks manually or by writing custom scripts.

An example of this comes when trying to create a managed database cluster. Ansible cannot do this so I wrote a Python script to handle database management.

https://github.com/seaburr/WordPressOnDigitalOcean/tree/master/roles/database-server

I do not feel DigitalOcean should fill the gaps either. Why? Because Ansible is a configuration management tool that ensures resources are configured in a desired state. Infrastructure creation is not Ansible’s job. There are specific tools for infrastructure creation… Enter Terraform.

Terraform is a tool for defining providers (like DigitalOcean or AWS) and the resources (like droplets, load balancers, etc.) that your environment requires. Terraforms intent is to compare your infrastructure to your desired state and make corrections to bring your resources into compliance. It is a different concern from HOW the infrastructure is configured.

Over the next few months, I’m going to migrate infrastructure concerns out of Ansible and into Terraform. In fact, I’ve already got a POC to share.

https://github.com/seaburr/Terraform-On-DO

This repository defines the new standard for infrastructure that I am aiming for.

Here’s a simple mockup of the goal:

I did try to use the built-in graph functionality of Terraform to show this but it came out looking like this:

I’ve got boxes full of Pepe!

Anyways, it’s a work in progress. I’ve run into what I believe is a bug with the DigitalOcean Terraform provider and I’ve already raised a ticket with them to get resolved.

Next time, let’s actually learn something and dig into a resource and the provider configuration.