One of the attractive characteristics of Kubernetes is how it can run pretty much anywhere – in the cloud, in the data center, on the edge, on your local machine and much more. Leveraging existing investments in datacenter resources can be logical when deciding where to place new Kubernetes clusters, and this post goes into automating this with Rancher and Terraform.

Primer

For this exercise the following is leveraged:

• Rancher 2.3
• vSphere 6.7
• Ubuntu 18.04 LTS

An Ubuntu VM will be created and configured into a template to spin up Kubernetes nodes.

Step 1 – Preparing a Ubuntu Server VM

In Rancher 2.3 Node templates for vSphere can leverage either of the following:

For the purposes of this demo, "Deploy from template" will be used, given its simplicity.

To create a new VM template, we must first create a VM. Right-click an appropriate object in vCenter and select "New Virtual Machine"

Select a source:

Give it a name:

Give it a home (compute):

Give it a home (storage):

Specify the VM hardware version:

Specify the guest OS:

Configure the VM properties, ensure the Ubuntu install CD is mounted:

After this, power up the VM and walk through the install steps. After which it can be turned into a template:

Rancher doesn’t have much in the way of requirements for the VM. For this install method a VM needs to have:

• Cloud-Init (Installed by default on Ubuntu 18.04).
• SSH connectivity (Rancher will provide its own SSH certificates as per Cloud-Init bootstrap) – Ensure SSH server has been installed.

A Note on Cloud-Init

For Vanilla Ubuntu Server installs, it uses Cloud-Init as part of the general Installation process. As such, cloud-init can not be re-invoked on startup by default. To get around this for templating purposes, the VM must be void of the existing cloud-init configuration prior to being turned into a template. To accomplish this, run the following:

sudo rm -rf /var/lib/cloud/instances

Before shutting down the VM and converting it into a template.

Constructing the Terraform Script

Now the VM template has been created it can be leveraged by a Terraform script:

Specify the provider: (Note – insecure = "true" Is required for vCenter servers leveraging an untrusted certificate, such as self-signed.

provider "rancher2" {
  api_url    = "https://rancher.virtualthoughts.co.uk"
  access_key = #ommited - reference a Terraform varaible/environment variable/secret/etc
  secret_key = #ommited - reference a Terraform varaible/environment variable/secret/etc
  insecure = "true"
}

Specify the Cloud Credentials:

# Create a new rancher2 Cloud Credential
resource "rancher2_cloud_credential" "vsphere-terraform" {
  name = "vsphere-terraform"
  description = "Terraform Credentials"
  vsphere_credential_config {
    username = "Terraform@vsphere.local"
    password = #ommited - reference a Terraform varaible/environment variable/secret/etc
    vcenter = "svr-vcs-01.virtualthoughts.co.uk"
  }
}

Specify the Node Template settings:

Note we can supply extra cloud-config options to further customise the VM, including adding additional SSH keys for users.

resource "rancher2_node_template" "vSphereTestTemplate" {
  name = "vSphereTestTemplate"
  description = "Created by Terraform"
  cloud_credential_id = rancher2_cloud_credential.vsphere-terraform.id
   vsphere_config {
   cfgparam = ["disk.enableUUID=TRUE"]
   clone_from = "/Homelab/vm/Ubuntu1804WithCloudInit"
   cloud_config = "#cloud-config\nusers:\n  - name: demo\n    ssh-authorized-keys:\n      - ssh-rsa [SomeKey]
   cpu_count = "4"
   creation_type = "template"
   disk_size = "20000"
   memory_size = "4096"
   datastore = "/Homelab/datastore/NFS-500"
   datacenter = "/Homelab"
   pool = "/Homelab/host/MGMT/Resources"
   network = ["/Homelab/network/VDS-MGMT-DEFAULT"]
   }
}

Specify the cluster settings:

resource "rancher2_cluster" "vsphere-test" {
  name = "vsphere-test"
  description = "Terraform created vSphere Cluster"
  rke_config {
    network {
      plugin = "canal"
    }
  }
}

Specify the Node Pool:

resource "rancher2_node_pool" "nodepool" {

  cluster_id =  rancher2_cluster.vsphere-test.id
  name = "all-in-one"
  hostname_prefix =  "vsphere-cluster-0"
  node_template_id = rancher2_node_template.vSphereTestTemplate.id
  quantity = 1
  control_plane = true
  etcd = true
  worker = true
}

After which the script can be executed.

What’s going on?

From a high level the following activities are being executed:

1. Rancher requests VM’s from vSphere using supplied Cloud Credentials.
2. vSphere clones the VM Templateeverywhere with the specified configuration parameters.
3. An ISO image is mounted to the VM, which contains certificates and configuration generated by Rancher in the cloud-init format.
4. Cloud-Init on startup reads this ISO image and applies the configuration.
5. Rancher builds the Kubernetes cluster by Installing Docker and pulling down the images.

After which, a shiny new cluster will be created!

In a previous post, I went through the process of configuring Pi-Hole within a Kubernetes cluster for the purpose of facilitating a network-wide adblocking. Although helpful, I wanted to augment this with DNS over HTTPS.

Complete manifests can be found here. Shout out to visibilityspots for the cloudflared image on Dockerhub

Why?

DNS, as a protocol, is insecure and can be prone to manipulation and man-in-the-middle attacks. DNS over HTTPS helps address this by encrypting the data between the DNS over HTTPS client and the DNS over HTTPS-based DNS resolver. One of which is provided by Cloudflare.

Thankfully, Pi-Hole has some documentation on how to implement this for the traditional Pi-Hole setups. But for Kubernetes-based deployments, this requires a different approach.

How?

The DNS over HTTPS client is facilitated by a Cloudflare daemon. In a traditional Pi-Hole setup this is simply run alongside Pi-Hole itself, but in a containerised environment there are predominantly two ways to address this:

As a separate microservice

This approach leverages two different deployments, one for the Pi-Hole service, and one for cloudflared. While workable, I felt that this was a less optimal approach

As another container within the Pi-Hole pod

Given the tight relationship between these containers, and the fact their respective services run on different ports, this seems like a more efficient approach.

As the containers share the same network interface, one pod can access the other over either the veth interface, or simply the localhost address. For Pi-Hole, we can facilitate this via a configmap change:

apiVersion: v1
kind: ConfigMap
metadata:
  name: pihole-env
  namespace: pihole-system
data:
  TZ: UTC
  DNS1: 127.0.0.1#5054
  DNS2: 127.0.0.1#5054

Testing

Once the respective manifest files have been deployed and clients are pointing to pi-hole as a DNS resolver, it can be tested by accessing https://1.1.1.1/help. As per the example below, DNS over HTTPS has been identified.

TLS Overview

If we take an example of accessing a website such as https://www.virtualthoughts.co.uk/, these are the high-level steps of what occurs:

1. The client initiates a connection to the web server requesting an HTTPS connection.
2. The web server responds with its public key. The client validates the key with its list of known Certificate Authorities.
3. A session key is generated by the client and encrypted with the web server’s public key and is sent back to the web server.
4. The web server decrypts the session key with its private key. End to end encryption is established.

By default, the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer.  For external, public-facing websites, this is an acceptable and well-established implementation of TLS. But what about communication between different microservices?

As opposed to monolithic applications, microservices are usually inter-connected which allow them to be scaled/modified/etc independently. But this does raise some challenges. For example:

• How do we ensure service-to-service communication is always encrypted?
• How can do we do this without changing the application source code?
• How can we automatically secure communication when we introduce a new service to an application?
• How can we authenticate clients and servers and fully establish a “zero trust” network policy?

Istio can help us address these challenges:

Example Application

To demonstrate Istio’s mTLS capabilities a WordPress Helm chart was deployed into a namespace with automatic sidecar injection. Installing and configuring Istio can be found on a previous blog post. By default, the policy specifies no mTLS between the respective services. As such, the topology of the solution is depicted below:

We can validate this by using Istioctl:

All of the “testsite” services (WordPress frontend and backend) Envoy proxies are using HTTP as their transport mechanism. Therefore mTLS has not been configured yet.

Creating Istio Objects – Policy and Destination Rules

As you might expect, establishing mutual TLS (mTLS) is a two-part process, First, we must configure the clients to leverage mTLS, as well as the servers. This is accomplished with Policy and Destination rules.

Policy (AKA – what I, the server, will accept)

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "default"
namespace: "wordpress-app"
spec:
peers:
- mtls: {}

This example policy strictly enforces only mTLS connections between services within the “wordpress-app” namespace

DestinationRule (AKA – what I, the client, will send out)

 apiVersion: "authentication.istio.io/v1alpha1"
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "vt-wordpress-mariadb"
namespace: "wordpress-app"
spec:
host: "*.wordpress-app.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL

This example enforces the use of mutual TLS when communicating with any service in the wordpress-app namespace. Applying these and re-running the previous istioctl command yields the following result:

This is accomplished largely due to Citadel – a component in the Istio control plane that manages certificate creation and distribution:

When mTLS is configured the traffic flow (from a high level) can be described as follows:

• Citadel provides certificates to the sidecar pods and manages them.
• WordPress pod creates a packet to query the MYSQL database.
  • WordPress Envoy sidecar pod intercepts this and establishes a connection to the destination sidecar pod and presents its certificate for authenticity.
• MYSQL Envoy sidecar pod receives a connection request, validates the client’s certificate and sends its own back.
• WordPress Envoy sidecar pod receives MYSQL’s certificate and checks it for authenticity.
• Both proxies are in agreement as to each other’s identity and establish an encrypted tunnel between the two.

This is what makes it “mutual” TLS. In effect, both services are presenting, inspecting and validating each other’s certificate as a prerequisite for service-to-service communication. This differs from a standard HTTPs site described earlier on where only the client was validating the server.

Additional Comments

Some additional observations I’ve made from this exercise

• If enforcing strict mTLS on a service that’s exposed externally from a load balancer, your clients will obviously need to send x509 certificates that can be validated by Citadel. A more flexible alternative to this is to employ an Istio gateway that provides TLS termination at the cluster boundary. This negates the need to provision x509 certs to each and every client, whilst maintaining mTLS within the cluster.
  
• Envoy sidecar pods can affect liveness probes and might require you to implement

   sidecarInjectorWebhook.rewriteAppHTTPProbe=true 

  upon installing Helm