Cloud on Virtualthoughts

ArgoCD - Ordering with ApplicationSets

Sun, 16 Nov 2025 00:00:00 +0000

In a previous post, I alluded to the use of ApplicationSets for my homelab deployments. I continue to leverage them, to the point I now have quite a number of applications managed by one:

ArgoCD (Itself)
Cert-Manager
Cilium
External-snapshotter
Gateway API CRD’s
Gateway API gateways
Homepage
Kanboard
Kubevirt
Longhorn
OpenTelemetry Operator
Sealed Secrets
System Upgrade Controller

The problem I had, was there was no ordering, dependency management or concurrency limits, so applications would simply update as and whenever changes were pushed.

KubeVirt on ARM64 - CDI Workaround

Mon, 07 Apr 2025 00:00:00 +0000

According to the KubeVirt documentation, CDI is not currently supported on ARM64, which is the architecture my Turing RK1 nodes use.

As a workaround, I experimented with writing an image directly to a PVC which can then be cloned/mounted to a KubeVirt VM. This example dd's an ISO image to a PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: fedora-workstation-pvc
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 30Gi
 volumeMode: Block
---
apiVersion: batch/v1
kind: Job
metadata:
 name: upload-fedora-workstation-job
spec:
 template:
 spec:
 containers:
 - name: writer
 image: fedora:latest
 command: ["/bin/bash", "-c"]
 args:
 - |
 set -e
 echo "[1/3] Installing tools..."
 dnf install -y curl xz
 echo "[2/3] Downloading and decompressing Fedora Workstation image..."
 curl -L https://download.fedoraproject.org/pub/fedora/linux/releases/41/Workstation/aarch64/images/Fedora-Workstation-41-1.4.aarch64.raw.xz | xz -d > /tmp/disk.raw
 echo "[3/3] Writing image to PVC block device..."
 dd if=/tmp/disk.raw of=/dev/vda bs=4M status=progress conv=fsync
 echo "Done writing Fedora Workstation image to PVC!"
 volumeDevices:
 - name: disk
 devicePath: /dev/vda
 volumeMounts:
 - name: tmp
 mountPath: /tmp
 securityContext:
 runAsUser: 0
 restartPolicy: Never
 volumes:
 - name: disk
 persistentVolumeClaim:
 claimName: fedora-workstation-pvc
 - name: tmp
 emptyDir: {}

Which can then be mounted to a VM:

Replicating my vSphere network configuration in Openshift Virtualisation

Mon, 05 Feb 2024 00:00:00 +0000

Red Hat Openshift Virtualisation provides a platform for running and managing Virtual Machines alongside Containers using a consistent API. It also provides a mechanism for migrating VMs from platforms such as vSphere.

As I have both environments, I wanted to deploy an Openshift Virtualisation setup that mimics my current vSphere setup so I could migrate Virtual Machines to it.

Existing vSphere Design

Below is a diagram depicting my current vSphere setup. My ESXi hosts are dual-homed with a separation of management (vmkernel) and virtual machine traffic.

Diving into an eBPF + Go Example: Part 1

Wed, 24 Jan 2024 00:00:00 +0000

Part 1 / Part 2 / Part 3

Preamble: In preparation for writing this I looked at some excellent content created by Liz Rice and Daniel Finneran - including videos, code and literature. I highly recommend checking out their work.

The Cilium eBPF documentation has some excellent examples of getting started with eBPF and Go. As a “hobbyist” programmer, I wanted to cement some of these concepts by digging deeper into one of the examples. Part of my learning style is to compile my own notes on a given topic, and this post is essentially that.

Diving into an eBPF + Go Example: Part 3 (Bonus Round)

Wed, 24 Jan 2024 00:00:00 +0000

Part 1 / Part 2 / Part 3

With one example explored, I wanted to put a spin on it - Therefore, I had an idea:

“Can I use eBPF to identify and store the contents of the protocol header for IP packets on a specific interface?”

It’s more of a rhetorical question - of course we can! The code can be found here.

To summarise, the eBPF C program is a little more complicated. It still leverages XDP, however instead of counting the number of packets, it will inspect each IP packet, extract the protocol number, and store it in a map.

Diving into an eBPF + Go Example: Part 2

Tue, 23 Jan 2024 00:00:00 +0000

Part 1 / Part 2 / Part 3

In Part 1, we had a look at creating the eBPF program in C, which we will need to compile into eBPF bytecode and inject into our Go application

Rather than copy/paste the exact instructions, the ebpf-go documentation outlines the process in the toolchain to create the scaffolding for the Go application.

package main

import (
 "log"
 "net"
 "os"
 "os/signal"
 "time"

 "github.com/cilium/ebpf/link"
 "github.com/cilium/ebpf/rlimit"
)

func main() {
 // Remove resource limits for kernels <5.11.
 if err := rlimit.RemoveMemlock(); err != nil {
 log.Fatal("Removing memlock:", err)
 }

 // Load the compiled eBPF ELF and load it into the kernel.
 var objs counterObjects
 if err := loadCounterObjects(&objs, nil); err != nil {
 log.Fatal("Loading eBPF objects:", err)
 }
 defer objs.Close()

 ifname := "eth0" // Change this to an interface on your machine.
 iface, err := net.InterfaceByName(ifname)
 if err != nil {
 log.Fatalf("Getting interface %s: %s", ifname, err)
 }

 // Attach count_packets to the network interface.
 link, err := link.AttachXDP(link.XDPOptions{
 Program: objs.CountPackets,
 Interface: iface.Index,
 })
 if err != nil {
 log.Fatal("Attaching XDP:", err)
 }
 defer link.Close()

 log.Printf("Counting incoming packets on %s..", ifname)

 // Periodically fetch the packet counter from PktCount,
 // exit the program when interrupted.
 tick := time.Tick(time.Second)
 stop := make(chan os.Signal, 5)
 signal.Notify(stop, os.Interrupt)
 for {
 select {
 case <-tick:
 var count uint64
 err := objs.PktCount.Lookup(uint32(0), &count)
 if err != nil {
 log.Fatal("Map lookup:", err)
 }
 log.Printf("Received %d packets", count)
 case <-stop:
 log.Print("Received signal, exiting..")
 return
 }
 }
}

As there’s already an example, let’s dig into the prominent sections:

Changing the default apps wildcard certificate in OCP4

Sat, 30 Dec 2023 00:00:00 +0000

In a standard OCP4 installation, several route objects are created by default and secured with a internally signed wildcard certificate.

These routes are configured as <app-name>.apps.<domain>. In my example, I have a cluster with the assigned domain ocp-acm.virtualthoughts.co.uk, which results in the routes below:

oauth-openshift.apps.ocp-acm.virtualthoughts.co.uk
console-openshift-console.apps.ocp-acm.virtualthoughts.co.uk
grafana-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk
thanos-querier-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk
prometheus-k8s-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk
alertmanager-main-openshift-monitoring.apps.ocp-acm.virtualthoughts.co.uk

Inspecting console-openshift-console.apps.ocp-acm.virtualthoughts.co.uk shows us the default wildcard TLS certificate used by the Ingress Operator:

Because it’s internally signed, it’s not trusted by default by external clients. However, this can be changed.

Improving the CI/build process for the community Rancher Exporter

Mon, 05 Jun 2023 00:00:00 +0000

One of my side projects is developing and maintaining an unofficial Prometheus Exporter for Rancher. It exposes metrics pertaining to Rancher-specific resources including, but not limited to managed clusters, Kubernetes versions, and more. Below shows an example dashboard based on these metrics.

Incidentally, if you are using Rancher, I’d love to hear your thoughts/feedback.

Previous CI workflow

The flowchart below outlines the existing process. Whilst automated, pushing directly to latest is bad practice.

Debugging cloud-init not executing runcmd commands

Wed, 18 Jan 2023 00:00:00 +0000

Background

Rancher leverages cloud-init for the provisioning of Virtual Machines on a number of infrastructure providers, as below:

I recently encountered an issue whereby vSphere based clusters using an Ubuntu VM template would successfully provision, but SLES based VM templates would not.

What does Rancher use cloud-init for?

This is covered in the Masterclass session I co-hosted, but as a refresher, particularly with the vSphere driver, Rancher will mount an ISO image to the VM to deliver the user-data portion of a cloud-init configuration. The contents of which look like this:

Evaluating Harvester in vSphere

Mon, 20 Dec 2021 00:00:00 +0000

Disclaimer - The use of nested virtualisation is not a supported topology

Harvester is an open-source HCI solution aimed at managing Virtual Machines, similar to vSphere and Nutanix, with key differences including (but not limited to):

Fully Open Source
Leveraging Kubernetes-native technologies
Integration with Rancher

Testing/evaluating any hyperconverged solution can be difficult - It usually requires having dedicated hardware as these solutions are designed to work directly on bare metal. However, we can circumvent this by leveraging nested virtualisation - something which may be familiar with a lot of homelabbers (myself included) - which involves using an existing virtualisation solution provision workloads that also leverage virtualisation technology.

Creating Kubernetes Clusters with Rancher and Pulumi

Thu, 13 May 2021 00:00:00 +0000

tldr; Here is the code repo

Intro

My Job at Suse (via Rancher) involves hosting a lot of demos, product walk-throughs and various other activities that necessitate spinning up tailored environments on-demand. To facilitate this, I previously leaned towards Terraform, and have since curated a list of individual scripts I have to manage on an individual basis as they address a specific use case.

This approach reached a point where it became difficult to manage. Ideally, I wanted an IaC environment that catered for:

K3s, Rancher and Pulumi

Tue, 06 Apr 2021 00:00:00 +0000

TLDR; Repo can be found here (Be warned, I’m at best, a hobbyist programmer and certainly not a software engineer in my day job)

I’ve been recently getting acquainted with Pulumi as an alternative to Terraform for managing my infrastructure. I decided to create a repo that would do a number of activities to stand up Rancher in a new K3s cluster, all managed by Pulumi in my vSphere Homelab, consisting of the following activities:

Rancher, vSphere Network Protocol Profiles and static IP addresses for k8s nodes [Updated 2023]

Sun, 29 Mar 2020 00:00:00 +0000

Edit: This post has been updated to reflect changes in newer versions of Rancher.

Note: As mentioned by Jonathan in the comments, disabling cloud-init’s initial network configuration is recommended. To do this, create a file:

/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

To contain:

network: {config: disabled}

In your VM template.

How networking configuration is applied to k8s nodes (or VM’s in general) in on-premises environments is usually achieved by one of two ways - DHCP or static. For some, DHCP is not a popular option and static addresses can be time-consuming to manage, particularly when there’s no IPAM feature in Rancher. In this blog post I go through how to leverage vSphere Network Protocol Profiles in conjunction with Rancher and Cloud-Init to reliably, and predictably apply static IP addresses to deployed nodes.

On-prem K8s clusters with Rancher, Terraform and Ubuntu

Fri, 27 Dec 2019 00:00:00 +0000

One of the attractive characteristics of Kubernetes is how it can run pretty much anywhere - in the cloud, in the data center, on the edge, on your local machine and much more. Leveraging existing investments in datacenter resources can be logical when deciding where to place new Kubernetes clusters, and this post goes into automating this with Rancher and Terraform.

Primer

For this exercise the following is leveraged:

Rancher 2.3
vSphere 6.7
Ubuntu 18.04 LTS

An Ubuntu VM will be created and configured into a template to spin up Kubernetes nodes.

Pi-Hole and K8s v2 - Now with DNS over HTTPS

Thu, 07 Nov 2019 00:00:00 +0000

In a previous post, I went through the process of configuring Pi-Hole within a Kubernetes cluster for the purpose of facilitating a network-wide adblocking. Although helpful, I wanted to augment this with DNS over HTTPS.

Complete manifests can be found here. Shout out to visibilityspots for the cloudflared image on Dockerhub

Why?

DNS, as a protocol, is insecure and can be prone to manipulation and man-in-the-middle attacks. DNS over HTTPS helps address this by encrypting the data between the DNS over HTTPS client and the DNS over HTTPS-based DNS resolver. One of which is provided by Cloudflare.

Creating a highly available, cross AZ, loadbalanced ETCD cluster in AWS with Terraform

Sun, 08 Sep 2019 00:00:00 +0000

Having experimented with Terraform recently, I decided to leverage this tool by creating an etcd cluster in AWS. This blog post goes through the steps used to accomplish this. For readability, I’ve only quoted pertinent code snippets, but all of the code can be found at https://github.com/David-VTUK/terraformec2.

Disclaimer

I do not profess to be an Etcd, Terraform or AWS expert, therefore he be dragons in the form of implementations unlikely to be best practice or production-ready. In particular, I would like to revisit this at some point and enhance it to include:

Application security with mutual TLS (mTLS) via Istio

Mon, 15 Jul 2019 00:00:00 +0000

TLS Overview

If we take an example of accessing a website such as https://www.virtualthoughts.co.uk/, these are the high-level steps of what occurs:

The client initiates a connection to the web server requesting an HTTPS connection.
The web server responds with its public key. The client validates the key with its list of known Certificate Authorities.
A session key is generated by the client and encrypted with the web server’s public key and is sent back to the web server.
The web server decrypts the session key with its private key. End to end encryption is established.

By default, the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer. For external, public-facing websites, this is an acceptable and well-established implementation of TLS. But what about communication between different microservices?

Step by Step - Istio up and running

Sun, 23 Jun 2019 00:00:00 +0000

Service Mesh is a pretty hot topic in the Kubernetes ecosystem currently, and I wanted to get it up and running in my own lab environment. Istio’s documentation has a pre-baked solution to demonstrate some of its capabilities (a book app, if memory serves me correctly), but I wanted to deploy my own app to get more “hands-on” experience with the tech, even if it’s only very basic to start with.

Introducing Velero - Backup and DR for Kubernetes Applications

Wed, 12 Jun 2019 00:00:00 +0000

What is Velero?

Velero (previously known as Heptio ARK) provides a suite of tools to backup Kubernetes resources and applications for two main purposes:

Disaster Recovery - Recover Kubernetes cluster components and applications.
Migration - Migrate your Kubernetes applications to another Kubernetes cluster.

Migrating Kubernetes applications is a compelling use case. One of the significant benefits of using Kubernetes is the predictability of the platform and consequently the portability of applications that reside on it. With the main exception of nuances with persistent storage, the Kubernetes API will fell almost indistinguishable whether it resides on prem, GKE, AKS, EKS and elsewhere. If you have your Kubernetes-based application on one provider and want to migrate it to another or duplicate it to run elsewhere for dev/test, this can easily be achieved. Especially with Velero.

Bootstrapping Prometheus, Grafana and Alertmanager to PKS deployed K8s Clusters

Tue, 14 May 2019 00:00:00 +0000

PKS is a comprehensive platform for the provisioning and management of Kubernetes clusters, which can be further enhanced by leveraging its extensibility options. In this post, we will modify a plan to deploy a yaml manifest file which provisions Prometheus, Grafana, and Alertmanager backed by NSX-T load balancers.

Why Prometheus, Grafana and Alertmanager?

The Cloud Native Computing Foundation accepted Prometheus as its second incubated project, the first being Kubernetes. Originally developed by SoundCloud. It has quickly become a popular platform for the monitoring of Kubernetes platforms. Built upon a powerful analytics engine, extensive and highly flexible data modeling can be accomplished with relative ease.

CKA Exam Experience (Inc study & lab guide)

Tue, 07 May 2019 00:00:00 +0000

Introduction

Over the long bank holiday weekend, I sat and passed the Certified Kubernetes Exam (CKA). This blog post goes over my experience (With respect to the NDA) together with a lab guide I’ve made which I’ve uploaded hoping it might help others.

Format

The online exams consist of a set of performance-based items (problems) to be solved on the command line. For the CKA there are 24 questions of varying difficulty. At the time of writing, the only option to sit this exam is through remote proctoring.

Exposing the K8s dashboard via a NSX-T Load balancer

Mon, 15 Apr 2019 00:00:00 +0000

For the following to work, your k8s infrastructure needs to leverage some kind of CNI that’s able to provision load balancers. For this example I’m leveraging PKS which has native integration with NSX-T.

The default way to access the Kubernetes dashboard is to leverage the kubectl proxy command. However, this is somewhat limiting for a production environment. An alternative way is to expose the dashboard through a load balancer.

PKS, Harbor and the importance of container registries

Wed, 27 Feb 2019 00:00:00 +0000

What are container registries and why do we need them?

A lot of the time, particularly when individuals and organisations are evaluating, testing and experimenting with containers they will use public container registries such as Docker Hub. These public registries provide an easy-to-use, simple way to access images. As developers, application owners, system admins etc gain familiarity and experience additional operational considerations need to be explored, such as:

Organisation - How can we organise container images in a meaningful way? Such as by environment state (Prod/Dev/Test) and application type?
RBAC - How can we implement role-based access control to a container registry?
Vulnerability Scanning - How can we scan container images for known vulnerabilities?
Efficiency - How can we centrally manage all our container images and deploy an application from them?
Security - Some images need to kept under lock and key, rather than using an external service like Docker Hub.

Introducing VMware Harbor Registry

VMware Harbor Registry has been designed to address these considerations as enterprise-class container registry solution with integration into PKS. In this post, We’ll have a quick primer on getting up and running with Harbor in PKS and explore some of its features. To begin, we need to download PKS Harbor from the Pivotal site and import it into ops manager.

Kubernetes zero to hero - from single VM webserver to a scalable microservices infrastructure

Fri, 13 Jul 2018 00:00:00 +0000

Preamble

Having spent a number of months familiarising myself with container technology I inevitably got “stuck in” with Kubernetes. Containers are brilliant, but I personally don’t see the value of managing individual containers - it’s still the pets vs cattle mentality. Orchestrating containers with the likes of Kubernetes, however, makes a ton of sense and reinforces the microservices approach to building and deploying applications.

To test myself, I decided to document end-to-end the entire journey from taking a web server residing on a standalone virtual machine, containerise it, and deploying it via Kubernetes.

Introducing VMware Kubernetes Engine

Wed, 27 Jun 2018 00:00:00 +0000

On the 26th of June 2018, VMware publically announced VKE - VMware Kubernetes Engine in Beta (with GA planned for later on this year). For me, the development of this solution flew under the radar, and its subsequent release came as quite a surprise - albeit quite a good one. So, where exactly does this solution fit with other Kubernetes based solutions that currently exist?

VKE Overview

VKE sits within VMware’s portfolio of cloud-native solutions as is pitched as a fully managed, Kubernetes-as-a-service offering. Therefore we have multiple ways we can consume Kubernetes resources from the VMware ecosystem, depicted in the diagram below.

Hybrid Cloud monitoring with VMware vRealize Operations

Tue, 12 Jun 2018 00:00:00 +0000

Applications and the underlying infrastructure, be it public, private or hybrid cloud are becoming increasingly sophisticated. Because of this, the way in which we monitor and observe these environments requires more sophisticated tools. In this blog post, we look at vRealize Operations and how it can be a facilitator of true hybrid cloud monitoring.

What is vRealize Operations?

vRealize Operations forms part of the overall vRealize suite from VMware – a collection of products targeted to accommodate cloud management and automation. In particular, vRealize Operations, as the name implies, primarily caters to operations management with full visibility across physical, virtual and cloud-based environments. The anatomy of vRealize Operations is depicted below

GCP Kubernetes & VMware Wavefront - a practical demonstration

Mon, 04 Jun 2018 00:00:00 +0000

Wavefront

Back in 2017, VMware acquired Wavefront - a company based in the US which focuses predominantly on real-time metrics and monitoring of a really…really vast array of platforms and technologies. We have technologies that aid in adopting and promoting cloud-native implementations, but monitoring, in some peoples eyes, can be a bit of an afterthought. Wavefront to the rescue. Having developed some Kubernetes and Docker knowledge myself, it seemed rather fitting to get an example going.

Serverless and Containers - from a former ops guy

Fri, 11 May 2018 00:00:00 +0000

Post-AWS Summit 2018 Thoughts on Serverless and Containers

I was lucky enough to attend the AWS summit in London in May 2018. It was a first for me, and the experience was pretty awesome. With a veritable smorgasbord of chalk talks, instructor-led demos and vendor presence there was something for everyone. I gravitated towards the docker/lambda sessions as I had recently picked up learning container technology, which got me thinking - from my perspective (previous ops-centric), how does container technology compare to the likes of serverless? When would you use one over the other? Whilst on the train home from London I decided to jot down my notes into this post.