Target topology

Recommended first Kubernetes tenant cluster:

OpenStack Project: k8s-lab

Network:
  public1              Existing OpenStack external/provider network
  k8s-net              Private Kubernetes tenant network
  k8s-subnet           10.20.0.0/24
  k8s-router           Router to public1

VMs:
  k8s-cp-01            Control plane
  k8s-worker-01        Worker
  k8s-worker-02        Worker
  k8s-gpu-01           Optional later GPU worker

Kubernetes:
  Pod CIDR:            10.244.0.0/16
  Service CIDR:        10.96.0.0/12
  CNI:                 Cilium
  Ingress:             ingress-nginx or Cilium ingress
  LB learning path:    MetalLB
  Storage:             Ceph CSI if Ceph is reachable
  Observability:       kube-prometheus-stack + Loki + Tempo + Mimir/Grafana

Assumptions:

OpenStack VIP:         192.168.1.50
OpenStack admin RC:    /etc/kolla/admin-openrc.sh
External network:      public1
Floating IP range:     already configured in OpenStack
Tenant VM OS:          Ubuntu 24.04 cloud image
Kubernetes version:    use one supported version consistently across all nodes

---

Step 1 — Create a Kubernetes Project in OpenStack

Load admin credentials on ctrl:

source /etc/kolla/admin-openrc.sh

Create a dedicated project:

openstack project create k8s-lab \
  --description "Tenant project for Kubernetes on OpenStack homelab"

Create a user:

openstack user create k8s-admin \
  --project k8s-lab \
  --password-prompt

Give the user project admin rights:

openstack role add \
  --project k8s-lab \
  --user k8s-admin \
  admin

Optional but useful: create a normal member user later for testing:

openstack user create k8s-user \
  --project k8s-lab \
  --password-prompt

openstack role add \
  --project k8s-lab \
  --user k8s-user \
  member

Check:

openstack project list
openstack user list
openstack role assignment list --project k8s-lab --names

Create a project RC file:

cat > ~/k8s-lab-openrc.sh <<'EOF'
export OS_AUTH_URL=http://192.168.1.50:5000/v3
export OS_PROJECT_NAME=k8s-lab
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USERNAME=k8s-admin
export OS_USER_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_INTERFACE=public
export OS_REGION_NAME=RegionOne
echo "Enter OpenStack password for k8s-admin:"
read -sr OS_PASSWORD_INPUT
export OS_PASSWORD="$OS_PASSWORD_INPUT"
unset OS_PASSWORD_INPUT
EOF

chmod 600 ~/k8s-lab-openrc.sh

Use it:

source ~/k8s-lab-openrc.sh
openstack token issue

---

Step 2 — Build Networking

Create the Kubernetes tenant network:

openstack network create k8s-net

Create the subnet:

openstack subnet create k8s-subnet \
  --network k8s-net \
  --subnet-range 10.20.0.0/24 \
  --gateway 10.20.0.1 \
  --dns-nameserver 192.168.1.1

Create router:

openstack router create k8s-router

Attach router to external network:

openstack router set k8s-router --external-gateway public1

Attach Kubernetes subnet:

openstack router add subnet k8s-router k8s-subnet

Create security group:

openstack security group create k8s-sg \
  --description "Kubernetes tenant cluster security group"

Allow SSH:

openstack security group rule create k8s-sg \
  --protocol tcp \
  --dst-port 22 \
  --ingress

Allow ICMP:

openstack security group rule create k8s-sg \
  --protocol icmp \
  --ingress

Allow Kubernetes API from your LAN or admin host. For lab simplicity:

openstack security group rule create k8s-sg \
  --protocol tcp \
  --dst-port 6443 \
  --ingress

Allow node-to-node traffic inside the project subnet:

openstack security group rule create k8s-sg \
  --protocol tcp \
  --remote-ip 10.20.0.0/24 \
  --ingress

openstack security group rule create k8s-sg \
  --protocol udp \
  --remote-ip 10.20.0.0/24 \
  --ingress

openstack security group rule create k8s-sg \
  --protocol icmp \
  --remote-ip 10.20.0.0/24 \
  --ingress

Cilium with VXLAN typically uses UDP 8472; Kubernetes also needs kubelet TCP 10250. Add explicit rules:

openstack security group rule create k8s-sg \
  --protocol udp \
  --dst-port 8472 \
  --remote-ip 10.20.0.0/24 \
  --ingress

openstack security group rule create k8s-sg \
  --protocol tcp \
  --dst-port 10250 \
  --remote-ip 10.20.0.0/24 \
  --ingress

Verify:

openstack network list
openstack subnet list
openstack router list
openstack security group rule list k8s-sg

---

Step 3 — Upload Kubernetes Images

Download an Ubuntu cloud image on ctrl:

mkdir -p ~/openstack-images
cd ~/openstack-images

wget -nc https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img

Upload to Glance:

source /etc/kolla/admin-openrc.sh

openstack image create ubuntu-24.04-k8s \
  --file ~/openstack-images/ubuntu-24.04-server-cloudimg-amd64.img \
  --disk-format qcow2 \
  --container-format bare \
  --public

Verify:

openstack image list

Recommended flavors:

openstack flavor create k8s.control.small \
  --ram 4096 \
  --disk 40 \
  --vcpus 2

openstack flavor create k8s.worker.medium \
  --ram 8192 \
  --disk 60 \
  --vcpus 4

openstack flavor create k8s.worker.large \
  --ram 16384 \
  --disk 80 \
  --vcpus 6

For your Dell T5500 homelab, start modestly:

Control plane: 2 vCPU / 4 GB RAM / 40 GB disk
Workers:       4 vCPU / 8 GB RAM / 60 GB disk
GPU node:      later, after PCI passthrough is stable

---

Step 4 — Build Kubernetes VMs

Create an SSH keypair:

source ~/k8s-lab-openrc.sh

ssh-keygen -t ed25519 -f ~/.ssh/k8s_openstack -C "k8s-openstack"

openstack keypair create \
  --public-key ~/.ssh/k8s_openstack.pub \
  k8s-key

Create a cloud-init file:

cat > ~/k8s-cloud-init.yaml <<'EOF'
#cloud-config
package_update: true
package_upgrade: false

users:
  - default

ssh_pwauth: false

packages:
  - qemu-guest-agent
  - curl
  - wget
  - vim
  - jq
  - net-tools
  - iproute2
  - ca-certificates
  - gnupg
  - lsb-release
  - apt-transport-https
  - software-properties-common

runcmd:
  - systemctl enable --now qemu-guest-agent
  - modprobe br_netfilter
  - modprobe overlay
  - |
    cat >/etc/modules-load.d/k8s.conf <<EOM
    overlay
    br_netfilter
    EOM
  - |
    cat >/etc/sysctl.d/99-kubernetes-cri.conf <<EOM
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward = 1
    EOM
  - sysctl --system
EOF

Create ports with fixed IPs. Fixed IPs make kubeadm and OpenStack CCM configuration easier:

openstack port create k8s-cp-01-port \
  --network k8s-net \
  --fixed-ip subnet=k8s-subnet,ip-address=10.20.0.11 \
  --security-group k8s-sg

openstack port create k8s-worker-01-port \
  --network k8s-net \
  --fixed-ip subnet=k8s-subnet,ip-address=10.20.0.21 \
  --security-group k8s-sg

openstack port create k8s-worker-02-port \
  --network k8s-net \
  --fixed-ip subnet=k8s-subnet,ip-address=10.20.0.22 \
  --security-group k8s-sg

Boot the VMs:

openstack server create k8s-cp-01 \
  --image ubuntu-24.04-k8s \
  --flavor k8s.control.small \
  --port k8s-cp-01-port \
  --key-name k8s-key \
  --user-data ~/k8s-cloud-init.yaml

openstack server create k8s-worker-01 \
  --image ubuntu-24.04-k8s \
  --flavor k8s.worker.medium \
  --port k8s-worker-01-port \
  --key-name k8s-key \
  --user-data ~/k8s-cloud-init.yaml

openstack server create k8s-worker-02 \
  --image ubuntu-24.04-k8s \
  --flavor k8s.worker.medium \
  --port k8s-worker-02-port \
  --key-name k8s-key \
  --user-data ~/k8s-cloud-init.yaml

Assign a floating IP to the control plane:

openstack floating ip create public1

CP_FIP=$(openstack floating ip list -f value -c "Floating IP Address" | head -1)

openstack server add floating ip k8s-cp-01 "$CP_FIP"

echo "$CP_FIP"

Optional: assign floating IPs to workers for easier initial setup:

openstack floating ip create public1
openstack floating ip create public1

openstack floating ip list

Then associate them manually:

openstack server add floating ip k8s-worker-01 <WORKER_01_FLOATING_IP>
openstack server add floating ip k8s-worker-02 <WORKER_02_FLOATING_IP>

Check:

openstack server list

SSH to the control plane:

ssh -i ~/.ssh/k8s_openstack ubuntu@"$CP_FIP"

---

Step 5 — Install Kubernetes

Do this on all Kubernetes VMs: k8s-cp-01, k8s-worker-01, k8s-worker-02.

Kubernetes requires a container runtime on every node; containerd is the straightforward CRI-compatible choice. The Kubernetes docs note that Kubernetes uses CRI-compatible runtimes and that dockershim was removed after Kubernetes 1.24.

5.1 Prepare all nodes

On each VM:

sudo swapoff -a

sudo sed -i '/ swap / s/^/#/' /etc/fstab

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

Install containerd:

sudo apt-get update

sudo apt-get install -y containerd

sudo mkdir -p /etc/containerd

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

sudo systemctl restart containerd
sudo systemctl enable containerd
sudo systemctl status containerd --no-pager

Install Kubernetes packages:

sudo apt-get update

sudo apt-get install -y apt-transport-https ca-certificates curl gpg

sudo mkdir -p /etc/apt/keyrings

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

The official kubeadm installation page currently documents Kubernetes v1.36; use the same minor version across all nodes unless you deliberately pin another supported series.

5.2 Initialise the control plane

On k8s-cp-01 only:

sudo kubeadm init \
  --control-plane-endpoint 10.20.0.11 \
  --apiserver-advertise-address 10.20.0.11 \
  --pod-network-cidr 10.244.0.0/16 \
  --service-cidr 10.96.0.0/12 \
  --upload-certs

Set up kubeconfig:

mkdir -p ~/.kube

sudo cp /etc/kubernetes/admin.conf ~/.kube/config

sudo chown "$(id -u):$(id -g)" ~/.kube/config

kubectl get nodes

Your node will show NotReady until Cilium is installed.

Get the worker join command:

kubeadm token create --print-join-command

Run the join command on each worker, for example:

sudo kubeadm join 10.20.0.11:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

Back on the control plane:

kubectl get nodes -o wide

---

Step 6 — Install Cilium

Cilium provides Kubernetes networking using eBPF and is commonly installed with Helm. The Cilium docs provide Helm-based installation guidance and require you to choose the datapath/IPAM mode appropriate to your environment.

Install Helm on k8s-cp-01:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Install Cilium CLI:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)

CLI_ARCH=amd64

curl -L --fail --remote-name-all \
  "https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz"

sudo tar xzvf "cilium-linux-${CLI_ARCH}.tar.gz" -C /usr/local/bin

rm "cilium-linux-${CLI_ARCH}.tar.gz"

Install Cilium with kube-proxy replacement disabled initially for the simplest first deployment:

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=false \
  --set ipam.mode=kubernetes \
  --set tunnelProtocol=vxlan

Wait:

kubectl -n kube-system rollout status ds/cilium
kubectl get nodes
cilium status --wait

Expected:

All nodes Ready
Cilium OK
CoreDNS Running

Optional Hubble:

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

kubectl -n kube-system rollout status deploy/hubble-relay
kubectl -n kube-system rollout status deploy/hubble-ui

---

Step 7 — Install Cloud Controller Manager

Kubernetes cloud-controller-manager links Kubernetes to the cloud provider API and separates cloud-specific control logic from core Kubernetes. For OpenStack, use cloud-provider-openstack.

This gives Kubernetes awareness of OpenStack instances, node addresses, and, if configured with the right OpenStack services, load balancer and route integration.

7.1 Create OpenStack application credentials

From ctrl, using the k8s-lab user:

source ~/k8s-lab-openrc.sh

openstack application credential create k8s-ccm

Save:

id
secret

7.2 Create cloud.conf

On k8s-cp-01:

mkdir -p ~/openstack-ccm

cat > ~/openstack-ccm/cloud.conf <<'EOF'
[Global]
auth-url=http://192.168.1.50:5000/v3
application-credential-id=<APP_CRED_ID>
application-credential-secret=<APP_CRED_SECRET>
region=RegionOne
domain-name=Default
tenant-name=k8s-lab

[LoadBalancer]
use-octavia=false

[BlockStorage]
bs-version=v3
EOF

Because your Kolla config did not enable Octavia, use-octavia=false is important. OpenStack CCM LoadBalancer services will not behave like a public cloud without Octavia. You will use Ingress/NodePort/MetalLB for lab exposure instead.

Create the secret:

kubectl -n kube-system create secret generic cloud-config \
  --from-file=cloud.conf=~/openstack-ccm/cloud.conf

7.3 Deploy OpenStack CCM

Create manifest:

cat > ~/openstack-ccm/openstack-ccm.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: openstack-cloud-controller-manager
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: openstack-cloud-controller-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
  - kind: ServiceAccount
    name: openstack-cloud-controller-manager
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: openstack-cloud-controller-manager
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: openstack-cloud-controller-manager
  template:
    metadata:
      labels:
        k8s-app: openstack-cloud-controller-manager
    spec:
      hostNetwork: true
      serviceAccountName: openstack-cloud-controller-manager
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
        - key: node.cloudprovider.kubernetes.io/uninitialized
          value: "true"
          effect: NoSchedule
      containers:
        - name: openstack-cloud-controller-manager
          image: registry.k8s.io/provider-os/openstack-cloud-controller-manager:latest
          args:
            - /bin/openstack-cloud-controller-manager
            - --v=2
            - --cloud-provider=openstack
            - --cloud-config=/etc/kubernetes/cloud.conf
            - --use-service-account-credentials=true
          volumeMounts:
            - name: cloud-config
              mountPath: /etc/kubernetes
              readOnly: true
      volumes:
        - name: cloud-config
          secret:
            secretName: cloud-config
EOF

Apply:

kubectl apply -f ~/openstack-ccm/openstack-ccm.yaml

kubectl -n kube-system get pods -l k8s-app=openstack-cloud-controller-manager
kubectl get nodes -o wide

If the image tag changes in your environment, pin a known compatible release from the cloud-provider-openstack project instead of latest.

---

Step 8 — Install Ceph CSI

Ceph CSI provides Kubernetes CSI drivers for RBD and CephFS; the Ceph CSI project supports RBD, CephFS and Kubernetes sidecar deployment YAMLs. Ceph documentation describes using Ceph RBD with Kubernetes through ceph-csi for dynamic provisioning.

Important: this step assumes your Kubernetes VMs can route to Ceph MONs and that you have Ceph credentials. If your Ceph cluster is only on the OpenStack host side and not reachable from tenant VMs, fix routing/firewalling first.

8.1 On Ceph cluster: create pool and user

On a Ceph admin node:

ceph osd pool create k8s-rbd 32

rbd pool init k8s-rbd

ceph auth get-or-create client.k8s \
  mon 'profile rbd' \
  osd 'profile rbd pool=k8s-rbd' \
  mgr 'profile rbd pool=k8s-rbd' \
  -o ceph.client.k8s.keyring

Get values:

ceph fsid
ceph mon dump
ceph auth get-key client.k8s

You need:

clusterID = Ceph FSID
monitors  = MON IPs, e.g. 192.168.1.10:6789
userID    = k8s
userKey   = client.k8s key
pool      = k8s-rbd

8.2 Install Ceph CSI RBD with Helm

On k8s-cp-01:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm repo update

kubectl create namespace ceph-csi-rbd

Create values file:

cat > ~/ceph-csi-rbd-values.yaml <<'EOF'
csiConfig:
  - clusterID: "<CEPH_FSID>"
    monitors:
      - "<MON1_IP>:6789"
      - "<MON2_IP>:6789"
      - "<MON3_IP>:6789"

secret:
  create: true
  userID: "k8s"
  userKey: "<CEPH_CLIENT_K8S_KEY>"

storageClass:
  create: true
  name: ceph-rbd
  clusterID: "<CEPH_FSID>"
  pool: k8s-rbd
  imageFeatures: layering
  csi.storage.k8s.io/fstype: ext4
  reclaimPolicy: Delete
  allowVolumeExpansion: true
EOF

Install:

helm install ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --namespace ceph-csi-rbd \
  -f ~/ceph-csi-rbd-values.yaml

Check:

kubectl -n ceph-csi-rbd get pods
kubectl get storageclass

Test PVC:

cat > ~/test-rbd-pvc.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ceph-rbd
  resources:
    requests:
      storage: 1Gi
EOF

kubectl apply -f ~/test-rbd-pvc.yaml
kubectl get pvc

---

Step 9 — Install Ingress

Use ingress-nginx first because it is familiar and easy to test.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

kubectl create namespace ingress-nginx

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.service.type=NodePort

Check:

kubectl -n ingress-nginx get pods
kubectl -n ingress-nginx get svc

Get NodePorts:

kubectl -n ingress-nginx get svc ingress-nginx-controller

You can access via:

http://<node-floating-ip>:<http-nodeport>
https://<node-floating-ip>:<https-nodeport>

Later, after MetalLB or Octavia is working, change to:

helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --reuse-values \
  --set controller.service.type=LoadBalancer

---

Step 10 — Install MetalLB

MetalLB is designed for bare-metal Kubernetes load-balancing and supports manifest, Kustomize and Helm installation methods. Its own docs warn to check cloud compatibility; on OpenStack tenant networks, L2 mode may require Neutron port security and allowed-address-pair adjustments.

For your homelab, treat MetalLB as a learning step. Because the Kubernetes nodes are VMs on Neutron VXLAN/private networking, MetalLB L2 may not work until Neutron allows the VIP/MAC behavior.

10.1 Install MetalLB

helm repo add metallb https://metallb.github.io/metallb
helm repo update

kubectl create namespace metallb-system

helm install metallb metallb/metallb \
  --namespace metallb-system

Wait:

kubectl -n metallb-system get pods

10.2 Choose an internal LB range

Use unused IPs from k8s-subnet, for example:

10.20.0.200 - 10.20.0.220

Create MetalLB pool:

cat > ~/metallb-pool.yaml <<'EOF'
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: k8s-lb-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.20.0.200-10.20.0.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: k8s-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - k8s-lb-pool
EOF

kubectl apply -f ~/metallb-pool.yaml

10.3 OpenStack Neutron allowed address pairs

For each Kubernetes node port, allow the MetalLB range. On ctrl with OpenStack credentials:

source ~/k8s-lab-openrc.sh

openstack port list --server k8s-cp-01
openstack port list --server k8s-worker-01
openstack port list --server k8s-worker-02

For each port ID:

openstack port set <PORT_ID> \
  --allowed-address ip-address=10.20.0.200

openstack port set <PORT_ID> \
  --allowed-address ip-address=10.20.0.201

For a range, you may need to add individual IPs or disable port security for lab testing:

openstack port set <PORT_ID> --disable-port-security

For learning, disabling port security is acceptable; for production, prefer specific allowed-address-pairs.

---

Step 11 — Install Metrics

Install metrics-server:

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm repo update

kubectl create namespace metrics-server

helm install metrics-server metrics-server/metrics-server \
  --namespace metrics-server \
  --set args="{--kubelet-insecure-tls}"

Check:

kubectl -n metrics-server get pods
kubectl top nodes
kubectl top pods -A

If kubectl top works, scheduling labs and HPA tests become easier.

---

Step 12 — Install LGTM

For your observability stack, install:

Grafana
Loki
Tempo
Mimir or Prometheus
kube-prometheus-stack
Alloy or Promtail equivalent

For a first Kubernetes-on-OpenStack lab, use kube-prometheus-stack plus Loki and Tempo. You can add Mimir after Prometheus is working.

Create namespace:

kubectl create namespace observability

Install kube-prometheus-stack:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --set grafana.service.type=NodePort \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=ceph-rbd \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.accessModes[0]=ReadWriteOnce \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=20Gi

Install Loki:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install loki grafana/loki \
  --namespace observability \
  --set deploymentMode=SingleBinary \
  --set loki.auth_enabled=false \
  --set singleBinary.replicas=1

Install Grafana Alloy for log collection:

helm install alloy grafana/alloy \
  --namespace observability

Install Tempo:

helm install tempo grafana/tempo \
  --namespace observability

Check:

kubectl -n observability get pods
kubectl -n observability get svc

Expose Grafana:

kubectl -n observability get svc kube-prometheus-stack-grafana

If NodePort:

http://<node-floating-ip>:<grafana-nodeport>

Get Grafana password:

kubectl -n observability get secret kube-prometheus-stack-grafana \
  -o jsonpath="{.data.admin-password}" | base64 -d
echo

---

Step 13 — Deploy Sample Applications

Create namespace:

kubectl create namespace demo

Deploy nginx:

cat > ~/demo-nginx.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:stable
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: demo
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80
EOF

kubectl apply -f ~/demo-nginx.yaml

Test:

kubectl -n demo get pods -o wide
kubectl -n demo get svc

Ingress test:

cat > ~/demo-nginx-ingress.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: demo
spec:
  ingressClassName: nginx
  rules:
    - host: nginx.k8s.lab
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80
EOF

kubectl apply -f ~/demo-nginx-ingress.yaml
kubectl -n demo get ingress

On your workstation, add /etc/hosts pointing nginx.k8s.lab to the ingress NodePort/floating IP path.

---

Step 14 — Learn Kubernetes Scheduling

Label nodes:

kubectl label node k8s-worker-01 workload=general
kubectl label node k8s-worker-02 workload=observability

Deploy pod with nodeSelector:

cat > ~/schedule-node-selector.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scheduled-nginx
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: scheduled-nginx
  template:
    metadata:
      labels:
        app: scheduled-nginx
    spec:
      nodeSelector:
        workload: general
      containers:
        - name: nginx
          image: nginx:stable
EOF

kubectl apply -f ~/schedule-node-selector.yaml
kubectl -n demo get pods -o wide

Taint a node:

kubectl taint node k8s-worker-02 dedicated=observability:NoSchedule

Deploy toleration example:

cat > ~/schedule-toleration.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tolerated-nginx
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tolerated-nginx
  template:
    metadata:
      labels:
        app: tolerated-nginx
    spec:
      tolerations:
        - key: dedicated
          operator: Equal
          value: observability
          effect: NoSchedule
      nodeSelector:
        workload: observability
      containers:
        - name: nginx
          image: nginx:stable
EOF

kubectl apply -f ~/schedule-toleration.yaml
kubectl -n demo get pods -o wide

Remove taint later:

kubectl taint node k8s-worker-02 dedicated=observability:NoSchedule-

---

Step 15 — GPU Nodes

This is a later sub-phase because it requires OpenStack/Nova PCI passthrough first.

15.1 OpenStack side

On your OpenStack compute host that has the GPU, you need:

IOMMU enabled in BIOS
IOMMU enabled in Proxmox/OpenStack host kernel
GPU bound to vfio-pci
Nova configured for PCI passthrough
Flavor with PCI alias
VM scheduled onto GPU compute node

Create a GPU flavor after passthrough is working:

source /etc/kolla/admin-openrc.sh

openstack flavor create k8s.gpu.medium \
  --ram 16384 \
  --disk 80 \
  --vcpus 6

openstack flavor set k8s.gpu.medium \
  --property "pci_passthrough:alias"="gpu:1"

Boot GPU VM:

source ~/k8s-lab-openrc.sh

openstack server create k8s-gpu-01 \
  --image ubuntu-24.04-k8s \
  --flavor k8s.gpu.medium \
  --network k8s-net \
  --security-group k8s-sg \
  --key-name k8s-key \
  --user-data ~/k8s-cloud-init.yaml

15.2 Kubernetes side

Install GPU drivers on the VM, then NVIDIA container toolkit, then NVIDIA device plugin:

kubectl create namespace gpu-operator

For NVIDIA GPU Operator:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator

Check:

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
kubectl -n gpu-operator get pods

---

Step 16 — Install Cluster API

Cluster API is the next automation layer: it lets Kubernetes create and manage Kubernetes clusters declaratively.

Use a management cluster first. Your manually built cluster can act as the management cluster.

Install clusterctl:

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 \
  -o clusterctl

chmod +x clusterctl

sudo mv clusterctl /usr/local/bin/

Initialize core Cluster API:

clusterctl init

For OpenStack provider, you need environment variables and provider config. The Cluster API quickstart/provider documentation should be followed closely because provider variables change by release; use a pinned compatible cluster-api-provider-openstack release for repeatability.

Conceptually:

export OPENSTACK_AUTH_URL=http://192.168.1.50:5000/v3
export OPENSTACK_USERNAME=k8s-admin
export OPENSTACK_PASSWORD='<password>'
export OPENSTACK_PROJECT_NAME=k8s-lab
export OPENSTACK_DOMAIN_NAME=Default
export OPENSTACK_REGION=RegionOne
export OPENSTACK_CLOUD=openstack

Then:

clusterctl init --infrastructure openstack

Later goal:

Use Kubernetes manifests to create OpenStack-backed Kubernetes clusters.

This becomes your bridge from manually built clusters to self-service clusters.

---

Step 17 — Install ArgoCD

Argo CD is a declarative GitOps CD system. Its getting-started docs install it into an argocd namespace using the official manifests; for production, pin a release version instead of tracking the stable branch.

Install:

kubectl create namespace argocd

kubectl apply -n argocd \
  --server-side \
  --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Check:

kubectl -n argocd get pods

Expose with NodePort:

kubectl -n argocd patch svc argocd-server \
  -p '{"spec": {"type": "NodePort"}}'

Get NodePort:

kubectl -n argocd get svc argocd-server

Get initial password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d
echo

Access:

https://<node-floating-ip>:<argocd-nodeport>

Login:

User: admin
Password: output from secret

---

Step 18 — Install Crossplane

Crossplane installs into an existing Kubernetes cluster and can be installed using its published Helm chart. The Crossplane docs list Helm-based install as the standard path and require an actively supported Kubernetes version and Helm.

Install:

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

kubectl create namespace crossplane-system

helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system

Check:

kubectl -n crossplane-system get pods
kubectl get crds | grep crossplane

Install providers for learning:

cat > ~/crossplane-providers.yaml <<'EOF'
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-kubernetes:latest
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-helm
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-helm:latest
EOF

kubectl apply -f ~/crossplane-providers.yaml

Later you can add an OpenStack provider if you want Crossplane to directly provision OpenStack resources. For the internal developer platform path, start with Kubernetes and Helm providers.

---

Step 19 — Build an Internal Developer Platform

This is the platform engineering goal of Phase 8.

You now have:

OpenStack = IaaS substrate
Kubernetes = tenant workload platform
Cilium = pod networking
Ceph CSI = persistent storage
Ingress = HTTP entry point
MetalLB = lab load balancer model
Metrics/LGTM = observability
ArgoCD = GitOps delivery
Crossplane = platform API/control plane
Cluster API = Kubernetes cluster lifecycle

19.1 Create platform namespaces

kubectl create namespace platform-system
kubectl create namespace platform-apps
kubectl create namespace platform-observability
kubectl create namespace platform-dev

19.2 Define developer golden paths

Create three initial developer offerings:

1. Static website app
2. HTTP API app
3. Stateful app with PVC

Each should provide:

Git repository template
Dockerfile
Helm chart
ArgoCD Application
Ingress
Resource requests/limits
ServiceMonitor
Dashboard
Alerts

19.3 GitOps repository layout

Recommended Git repo:

platform-gitops/
  clusters/
    k8s-openstack-lab/
      namespaces/
      ingress/
      observability/
      storage/
      apps/
  apps/
    demo-nginx/
      Chart.yaml
      values.yaml
      templates/
    demo-api/
    demo-stateful/
  platform/
    crossplane/
    compositions/
    claims/
  argocd/
    root-app.yaml

19.4 ArgoCD app-of-apps

Create root app:

cat > ~/argocd-root-app.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://example.com/your/platform-gitops.git
    targetRevision: main
    path: clusters/k8s-openstack-lab
  destination:
    server: https://kubernetes.default.svc
    namespace: platform-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

kubectl apply -f ~/argocd-root-app.yaml

Replace repoURL with your actual Git repository.

19.5 Crossplane developer claim model

Example future abstraction:

apiVersion: platform.blusas.local/v1alpha1
kind: WebApp
metadata:
  name: demo-api
spec:
  image: registry.local/demo-api:v1
  replicas: 3
  ingress:
    host: demo-api.k8s.lab
  observability:
    dashboard: true
    alerts: true

The goal is that a developer asks for a WebApp, not a Deployment, Service, Ingress, ServiceMonitor, Grafana dashboard and alerts separately.

---

Phase 8 Validation Commands

Run these regularly.

OpenStack layer

source ~/k8s-lab-openrc.sh

openstack server list
openstack network list
openstack router list
openstack floating ip list
openstack security group list

Kubernetes layer

kubectl get nodes -o wide
kubectl get pods -A
kubectl get svc -A
kubectl get ingress -A
kubectl get pvc -A
kubectl top nodes

Cilium

cilium status
kubectl -n kube-system get pods -l k8s-app=cilium

Storage

kubectl get storageclass
kubectl get pvc -A

Ingress

kubectl -n ingress-nginx get svc
kubectl get ingress -A

Observability

kubectl -n observability get pods
kubectl -n observability get svc

GitOps and platform

kubectl -n argocd get applications
kubectl -n crossplane-system get pods
kubectl get providers.pkg.crossplane.io

---

Recommended Phase 8 Execution Order

Do not try to do all 19 steps in one run. Use this milestone order.

Milestone 1 — Kubernetes VMs boot

Complete:

Step 1
Step 2
Step 3
Step 4

Success:

openstack server list
ssh ubuntu@<control-plane-floating-ip>

Milestone 2 — Basic Kubernetes works

Complete:

Step 5
Step 6

Success:

kubectl get nodes
cilium status
kubectl get pods -A

Milestone 3 — OpenStack integration

Complete:

Step 7

Success:

kubectl get nodes -o wide
kubectl -n kube-system get pods | grep openstack

Milestone 4 — Storage and ingress

Complete:

Step 8
Step 9
Step 10

Success:

kubectl get storageclass
kubectl get pvc
kubectl get ingress -A

Milestone 5 — Observability and apps

Complete:

Step 11
Step 12
Step 13
Step 14

Success:

kubectl top nodes
kubectl -n observability get pods
kubectl -n demo get pods,svc,ingress

Milestone 6 — Advanced platform engineering

Complete:

Step 15
Step 16
Step 17
Step 18
Step 19

Success:

kubectl -n argocd get applications
kubectl -n crossplane-system get pods
clusterctl describe cluster <cluster-name>

Key Design Notes

Because your OpenStack deployment currently has Octavia disabled, Kubernetes Service type=LoadBalancer through OpenStack CCM will not behave like a managed public cloud load balancer. Use NodePort, Ingress on NodePort, or MetalLB as a lab exercise until Octavia is added.

Because your OpenStack deployment currently has Cinder disabled, Kubernetes persistent storage should come from Ceph CSI directly only if the tenant VMs can reach the Ceph MONs. If you later enable Cinder with Ceph RBD underneath, you can also test OpenStack Cinder CSI as the cloud-native OpenStack storage integration.

Phase 8 is complete when you can:

1. Create Kubernetes VMs through OpenStack
2. Build a kubeadm cluster
3. Install Cilium
4. Run pods across worker nodes
5. Expose apps through ingress
6. Provision persistent volumes
7. Observe the cluster in Grafana
8. Deploy apps through ArgoCD
9. Start modelling platform abstractions with Crossplane