Kubernetes on Virtualthoughts

ArgoCD - Ordering with ApplicationSets

Sun, 16 Nov 2025 00:00:00 +0000

In a previous post, I alluded to the use of ApplicationSets for my homelab deployments. I continue to leverage them, to the point I now have quite a number of applications managed by one:

ArgoCD (Itself)
Cert-Manager
Cilium
External-snapshotter
Gateway API CRD’s
Gateway API gateways
Homepage
Kanboard
Kubevirt
Longhorn
OpenTelemetry Operator
Sealed Secrets
System Upgrade Controller

The problem I had, was there was no ordering, dependency management or concurrency limits, so applications would simply update as and whenever changes were pushed.

KubeVirt on ARM64 - CDI Workaround

Mon, 07 Apr 2025 00:00:00 +0000

According to the KubeVirt documentation, CDI is not currently supported on ARM64, which is the architecture my Turing RK1 nodes use.

As a workaround, I experimented with writing an image directly to a PVC which can then be cloned/mounted to a KubeVirt VM. This example dd's an ISO image to a PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: fedora-workstation-pvc
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 30Gi
 volumeMode: Block
---
apiVersion: batch/v1
kind: Job
metadata:
 name: upload-fedora-workstation-job
spec:
 template:
 spec:
 containers:
 - name: writer
 image: fedora:latest
 command: ["/bin/bash", "-c"]
 args:
 - |
 set -e
 echo "[1/3] Installing tools..."
 dnf install -y curl xz
 echo "[2/3] Downloading and decompressing Fedora Workstation image..."
 curl -L https://download.fedoraproject.org/pub/fedora/linux/releases/41/Workstation/aarch64/images/Fedora-Workstation-41-1.4.aarch64.raw.xz | xz -d > /tmp/disk.raw
 echo "[3/3] Writing image to PVC block device..."
 dd if=/tmp/disk.raw of=/dev/vda bs=4M status=progress conv=fsync
 echo "Done writing Fedora Workstation image to PVC!"
 volumeDevices:
 - name: disk
 devicePath: /dev/vda
 volumeMounts:
 - name: tmp
 mountPath: /tmp
 securityContext:
 runAsUser: 0
 restartPolicy: Never
 volumes:
 - name: disk
 persistentVolumeClaim:
 claimName: fedora-workstation-pvc
 - name: tmp
 emptyDir: {}

Which can then be mounted to a VM:

Customising ArgoCD ApplicationSets with Template Patches

Fri, 21 Feb 2025 00:00:00 +0000

In a recent attempt to automate my homelab cluster (ref), I now manage all of my cluster applications using ArgoCD, including cilium. I also leverage [applicationSet](https://argo-cd.readthedocs.io/en/stable/user-guide/application-set/) objects in ArgoCD as an app-of-apps pattern.

After a Cilium update however, it would fail to sync:

One way to address this is to add `ServerSideApply=true` to the the resulting application manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: bootstrap-applications
 namespace: argocd
spec:
 goTemplate: true
 goTemplateOptions: ["missingkey=error"]
 generators:
 - git:
 repoURL: 'https://github.com/David-VTUK/turing-pi-automation.git'
 revision: HEAD
 directories:
 - path: 'argocd-apps/helm-charts/import-from-cluster-standup/*'
 template:
 metadata:
 name: '{{ .path.basename }}'
 spec:
 project: default
 source:
 repoURL: 'https://github.com/David-VTUK/turing-pi-automation.git'
 targetRevision: HEAD
 path: '{{ .path.path }}'
 helm:
 valueFiles:
 - values.yaml
 destination:
 server: 'https://kubernetes.default.svc'
 namespace: '{{ .path.basename }}'
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true
 - ServerSideApply=true

The downside to this, however, is all applications from this applicationset will inherit this value, which is less than ideal.

Kubernetes on RK1 / Turing Pi 2: Automation with Ansible, Cilium and Cert-Manager

Fri, 30 Aug 2024 00:00:00 +0000

TLDR: Take me to the Playbook

Note - This is just a high-level overview, I’ll likely follow up with a post dedicated on the CIlium/BGP configuration.

I’ve had my Turing Pi 2 board for a while now, and during that time I’ve struggled to decide which automation tooling to use to bootstrap K3s to it. However, I reached a decision to use Ansible. It’s not something I’m overly familiar with, but this would provide a good opportunity to learn by doing.

Changing the default apps wildcard certificate in OCP4

Sat, 30 Dec 2023 00:00:00 +0000

In a standard OCP4 installation, several route objects are created by default and secured with a internally signed wildcard certificate.

These routes are configured as <app-name>.apps.<domain>. In my example, I have a cluster with the assigned domain ocp-acm.virtualthoughts.co.uk, which results in the routes below:

oauth-openshift.apps.ocp-acm.virtualthoughts.co.uk
console-openshift-console.apps.ocp-acm.virtualthoughts.co.uk
grafana-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk
thanos-querier-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk
prometheus-k8s-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk
alertmanager-main-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk

Inspecting console-openshift-console.apps.ocp-acm.virtualthoughts.co.uk shows us the default wildcard TLS certificate used by the Ingress Operator:

Because it’s internally signed, it’s not trusted by default by external clients. However, this can be changed.

Improving the CI/build process for the community Rancher Exporter

Mon, 05 Jun 2023 00:00:00 +0000

One of my side projects is developing and maintaining an unofficial Prometheus Exporter for Rancher. It exposes metrics pertaining to Rancher-specific resources including, but not limited to managed clusters, Kubernetes versions, and more. Below shows an example dashboard based on these metrics.

Incidentally, if you are using Rancher, I’d love to hear your thoughts/feedback.

Previous CI workflow

The flowchart below outlines the existing process. Whilst automated, pushing directly to latest is bad practice.

Debugging cloud-init not executing runcmd commands

Wed, 18 Jan 2023 00:00:00 +0000

Background

Rancher leverages cloud-init for the provisioning of Virtual Machines on a number of infrastructure providers, as below:

I recently encountered an issue whereby vSphere based clusters using an Ubuntu VM template would successfully provision, but SLES based VM templates would not.

What does Rancher use cloud-init for?

This is covered in the Masterclass session I co-hosted, but as a refresher, particularly with the vSphere driver, Rancher will mount an ISO image to the VM to deliver the user-data portion of a cloud-init configuration. The contents of which look like this:

Evaluating Harvester in vSphere

Mon, 20 Dec 2021 00:00:00 +0000

Disclaimer - The use of nested virtualisation is not a supported topology

Harvester is an open-source HCI solution aimed at managing Virtual Machines, similar to vSphere and Nutanix, with key differences including (but not limited to):

Fully Open Source
Leveraging Kubernetes-native technologies
Integration with Rancher

Testing/evaluating any hyperconverged solution can be difficult - It usually requires having dedicated hardware as these solutions are designed to work directly on bare metal. However, we can circumvent this by leveraging nested virtualisation - something which may be familiar with a lot of homelabbers (myself included) - which involves using an existing virtualisation solution provision workloads that also leverage virtualisation technology.

Taking a Modular Approach to my Homelab with Pulumi

Wed, 17 Nov 2021 00:00:00 +0000

Architecture

After reviewing the key components of my lab environment, I translated these into the Pulumi stacks as illustrated in the diagram below. Pulumi has a blog post about the benefits of adopting multiple stacks and I found organising my homelab this way enables greater flexibility and organisation. I can also use stacks as a “template” to further build out my lab environment, for example, repeating the “Tools-Cluster” stack to add additional clusters.

Creating Kubernetes Clusters with Rancher and Pulumi

Thu, 13 May 2021 00:00:00 +0000

tldr; Here is the code repo

Intro

My Job at Suse (via Rancher) involves hosting a lot of demos, product walk-throughs and various other activities that necessitate spinning up tailored environments on-demand. To facilitate this, I previously leaned towards Terraform, and have since curated a list of individual scripts I have to manage on an individual basis as they address a specific use case.

This approach reached a point where it became difficult to manage. Ideally, I wanted an IaC environment that catered for:

K3s, Rancher and Pulumi

Tue, 06 Apr 2021 00:00:00 +0000

TLDR; Repo can be found here (Be warned, I’m at best, a hobbyist programmer and certainly not a software engineer in my day job)

I’ve been recently getting acquainted with Pulumi as an alternative to Terraform for managing my infrastructure. I decided to create a repo that would do a number of activities to stand up Rancher in a new K3s cluster, all managed by Pulumi in my vSphere Homelab, consisting of the following activities:

End to end automation with CircleCI and ArgoCD Part 1 - Overview

Wed, 24 Jun 2020 00:00:00 +0000

Part 1 of this 3 part series goes through the process of setting up a CI/CD pipeline leveraging CircleCI and ArgoCD. The overall architecture is depicted below:

CI/CD Delivery Pipeline

The steps that will be implemented/accommodated are:

Developer commits code to a GitHub repo that is monitored by CircleCI.
CircleCI will perform the following tasks on all commits into the master branch:
1. Test the code. In this example, we’re leveraging go test.
2. If Testing completes successfully, build (compile) the code.
3. If building the code completes successfully, construct a docker image to accommodate the service. Push this image to DockerHub
4. If creating the Docker Image completes successfully, construct a basic deployment YAML file including the tag of the image that was recently uploaded.
5. Commit the YAML file to a separate GitHub repo, monitored by ArgoCD.
Argo CD will deploy the YAML manifest:
1. Automatically into the Test cluster
2. With manual approval into the Prod cluster

Part 2 - CircleCI Configuration

End to end automation with CircleCI and ArgoCD Part 2 - CircleCI Configuration

Wed, 24 Jun 2020 00:00:00 +0000

CircleCI is a continuous integration technology that is capable of building extremely complex pipelines. Being cloud-hosted and offering a free tier makes it very easy to get up and running.

Simply navigate to CircleCI’s website and log in with either Github or Bitbucket.

Select your org:

Select your project. For me, it’s CircleCI-Webapp-CI (Feel free to fork it or leverage your own)

The next step is to create a CircleCI config file that will dictate the steps in the pipeline. CircleCI provides a template to work with:

End to end automation with CircleCI and ArgoCD Part 3 – ArgoCD

Wed, 24 Jun 2020 00:00:00 +0000

ArgoCD is a Continuous Delivery tool designed for Kubernetes, This will be used to take the generated YAML file from the CI process and apply it to two clusters.

In this section, the following part of the overall CI/CD pipeline being implemented is depicted below.

ArgoCD will monitor for changes in the Webapp-CD Github Repo.
All changes are automatically applied to the Test cluster.
All changes will be staged for manual approval to Prod cluster

Install ArgoCD

ArgoCD has extensive installation documentation here. For ease, a community Helm chart has also been created.

Kubernetes on Virtualthoughts

ArgoCD - Ordering with ApplicationSets

KubeVirt on ARM64 - CDI Workaround

Customising ArgoCD ApplicationSets with Template Patches

Kubernetes on RK1 / Turing Pi 2: Automation with Ansible, Cilium and Cert-Manager

Changing the default apps wildcard certificate in OCP4

Improving the CI/build process for the community Rancher Exporter

Previous CI workflow

Debugging cloud-init not executing runcmd commands

Background

What does Rancher use cloud-init for?

Evaluating Harvester in vSphere

Taking a Modular Approach to my Homelab with Pulumi

Architecture

Creating Kubernetes Clusters with Rancher and Pulumi

Intro

K3s, Rancher and Pulumi

End to end automation with CircleCI and ArgoCD Part 1 - Overview

End to end automation with CircleCI and ArgoCD Part 2 - CircleCI Configuration

Sign Up and Set-Up

End to end automation with CircleCI and ArgoCD Part 3 – ArgoCD

Install ArgoCD