OpenStack Commands

1. Load OpenStack environment

In your Kolla-Ansible deployment, you normally load the admin credentials from:

source /etc/kolla/admin-openrc.sh

Verify authentication:

openstack token issue

Purpose:
Confirms that your shell has valid Keystone credentials and can obtain a token.

Useful environment check:

env | grep OS_

Purpose:
Shows the active OpenStack authentication variables such as:

OS_AUTH_URL
OS_PROJECT_NAME
OS_USERNAME
OS_REGION_NAME
OS_INTERFACE
OS_IDENTITY_API_VERSION

---

2. Global CLI inspection commands

Show OpenStack client help

openstack help

Shows the top-level command groups.

openstack help server
openstack help network
openstack help volume
openstack help image

Shows subcommands for a specific resource type.

Show command-specific help

openstack help server create
openstack help network create
openstack help subnet create
openstack help router set
openstack help volume create

Purpose:
Essential when checking exact flags for your installed OpenStackClient version.

---

3. Identity / Keystone commands

Keystone provides authentication, projects, users, roles, domains, and service catalog data.

List projects

openstack project list

Purpose:
Shows all OpenStack projects / tenants.

Show project details

openstack project show admin

Purpose:
Displays project ID, domain, enabled state, and metadata.

Create a project

openstack project create lab-project \
  --description "Homelab test project"

Purpose:
Creates an isolated tenant/project for workloads.

List users

openstack user list

Create user

openstack user create labuser \
  --password 'StrongPasswordHere' \
  --project lab-project

List roles

openstack role list

Assign role to user in project

openstack role add \
  --project lab-project \
  --user labuser \
  member

Purpose:
Gives labuser access to lab-project.

Show effective role assignments

openstack role assignment list \
  --user labuser \
  --project lab-project \
  --names

Purpose:
Useful when debugging Horizon or CLI permission issues.

List domains

openstack domain list

List service catalog

openstack catalog list

Purpose:
Shows registered OpenStack services and endpoints.

List services

openstack service list

Purpose:
Confirms which services are registered in Keystone, for example:

keystone
nova
neutron
glance
placement
cinder

List endpoints

openstack endpoint list

Filter for one service:

openstack endpoint list | grep -i nova
openstack endpoint list | grep -i neutron
openstack endpoint list | grep -i glance
openstack endpoint list | grep -i placement

Purpose:
Useful for diagnosing broken API routing, VIP issues, HAProxy issues, or incorrect public/internal/admin endpoint URLs.

---

4. Image / Glance commands

Glance stores VM images such as Ubuntu cloud images, Cirros, Rocky, Debian, etc.

List images

openstack image list

Show image details

openstack image show ubuntu-24.04

Upload a QCOW2 image

openstack image create ubuntu-24.04 \
  --file noble-server-cloudimg-amd64.img \
  --disk-format qcow2 \
  --container-format bare \
  --public

Purpose:
Uploads an image into Glance for VM creation.

Upload image as private

openstack image create ubuntu-private \
  --file ubuntu.img \
  --disk-format qcow2 \
  --container-format bare \
  --private

Set image properties

openstack image set ubuntu-24.04 \
  --property os_distro=ubuntu \
  --property os_version=24.04 \
  --property hw_scsi_model=virtio-scsi \
  --property hw_disk_bus=scsi

Purpose:
Improves Nova scheduling and guest device behaviour.

Mark image protected

openstack image set ubuntu-24.04 --protected

Purpose:
Prevents accidental deletion.

Delete image

openstack image delete ubuntu-private

---

5. Flavor commands

Flavors define VM sizing: vCPU, RAM, disk, extra specs.

List flavors

openstack flavor list

Create a basic flavor

openstack flavor create m1.small \
  --vcpus 2 \
  --ram 4096 \
  --disk 40

Create a GPU flavor

openstack flavor create gpu.1x \
  --vcpus 4 \
  --ram 8192 \
  --disk 80

Add GPU resource request to flavor

openstack flavor set gpu.1x \
  --property resources:VGPU=1

Or for PCI passthrough-style setups, depending on your Nova configuration:

openstack flavor set gpu.1x \
  --property pci_passthrough:alias='gpu:1'

Purpose:
Tells Nova that this flavor requires a GPU resource or PCI passthrough alias.

Show flavor extra specs

openstack flavor show gpu.1x

Delete flavor

openstack flavor delete gpu.1x

---

6. Keypair commands

List keypairs

openstack keypair list

Create a keypair from existing public key

openstack keypair create \
  --public-key ~/.ssh/id_ed25519_kolla.pub \
  kolla-key

Generate a new keypair and save private key

openstack keypair create lab-key > lab-key.pem
chmod 600 lab-key.pem

Show keypair fingerprint

openstack keypair show kolla-key

Delete keypair

openstack keypair delete lab-key

---

7. Networking / Neutron commands

Neutron manages tenant networks, provider networks, routers, ports, floating IPs, security groups and DHCP namespaces.

OpenStack Networking uses the Neutron API. The Networking API is REST-based and exposes network, subnet, router, port and security-group resources.

---

7.1 List networks

openstack network list

Show network

openstack network show private

Create tenant network

openstack network create private

Purpose:
Creates an isolated overlay network for project VMs.

Create subnet

openstack subnet create private-subnet \
  --network private \
  --subnet-range 10.10.10.0/24 \
  --gateway 10.10.10.1 \
  --dns-nameserver 1.1.1.1 \
  --dns-nameserver 8.8.8.8

Purpose:
Creates IPAM and DHCP configuration for the tenant network.

List subnets

openstack subnet list

Show subnet

openstack subnet show private-subnet

---

8. Provider / external network commands

For your setup, this is normally the network that maps to your physical LAN through OpenStack external networking.

Create provider external network

Example using flat provider network:

openstack network create public \
  --external \
  --provider-network-type flat \
  --provider-physical-network physnet1

Purpose:
Creates an external Neutron network backed by your physical provider network.

Create external subnet

Example using your LAN-style addressing:

openstack subnet create public-subnet \
  --network public \
  --subnet-range 192.168.1.0/24 \
  --allocation-pool start=192.168.1.200,end=192.168.1.240 \
  --gateway 192.168.1.1 \
  --dns-nameserver 1.1.1.1 \
  --no-dhcp

Purpose:
Defines the floating IP pool and external gateway.

Confirm external flag

openstack network show public -c router:external -c provider:network_type -c provider:physical_network

---

9. Router commands

Create router

openstack router create lab-router

Attach tenant subnet to router

openstack router add subnet lab-router private-subnet

Set external gateway

openstack router set lab-router \
  --external-gateway public

Purpose:
Allows instances on the private network to reach outside networks and allows floating IP routing.

Show router

openstack router show lab-router

List router ports

openstack port list --router lab-router

Remove router gateway

openstack router unset lab-router --external-gateway

Remove subnet from router

openstack router remove subnet lab-router private-subnet

---

10. Security group commands

Security groups are virtual firewall rules applied to instance ports.

List security groups

openstack security group list

Show default security group

openstack security group show default

Create security group

openstack security group create ssh-icmp \
  --description "Allow SSH and ICMP"

Allow SSH

openstack security group rule create ssh-icmp \
  --protocol tcp \
  --dst-port 22 \
  --remote-ip 0.0.0.0/0

Allow ping

openstack security group rule create ssh-icmp \
  --protocol icmp \
  --remote-ip 0.0.0.0/0

Allow HTTP

openstack security group rule create web \
  --protocol tcp \
  --dst-port 80 \
  --remote-ip 0.0.0.0/0

Allow HTTPS

openstack security group rule create web \
  --protocol tcp \
  --dst-port 443 \
  --remote-ip 0.0.0.0/0

List rules

openstack security group rule list ssh-icmp

---

11. Port commands

Ports are Neutron interfaces. Every VM NIC is a Neutron port.

List ports

openstack port list

Show port

openstack port show <port-id>

List ports on a network

openstack port list --network private

List ports for a server

openstack port list --server gpu-test-01

Create a fixed-IP port

openstack port create gpu-test-port \
  --network private \
  --fixed-ip subnet=private-subnet,ip-address=10.10.10.36 \
  --security-group ssh-icmp

Purpose:
Useful when you want deterministic VM IP assignment.

Boot server with specific port

openstack server create gpu-test-01 \
  --flavor gpu.1x \
  --image ubuntu-24.04 \
  --key-name kolla-key \
  --port gpu-test-port

---

12. Floating IP commands

List floating IPs

openstack floating ip list

Allocate floating IP

openstack floating ip create public

Attach floating IP to server

openstack server add floating ip gpu-test-01 192.168.1.201

Remove floating IP from server

openstack server remove floating ip gpu-test-01 192.168.1.201

Delete floating IP

openstack floating ip delete 192.168.1.201

---

13. Compute / Nova server commands

Nova manages instances. The openstack server command family is your main operational interface.

List servers

openstack server list

Show all projects:

openstack server list --all-projects

Long format:

openstack server list --long

Show server details

openstack server show gpu-test-01

Useful fields only:

openstack server show gpu-test-01 \
  -c id \
  -c name \
  -c status \
  -c addresses \
  -c flavor \
  -c image \
  -c OS-EXT-SRV-ATTR:host \
  -c OS-EXT-STS:power_state \
  -c OS-EXT-AZ:availability_zone

Purpose:
Shows placement host, power state, IPs and availability zone.

Create a VM

openstack server create test-vm-01 \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name kolla-key \
  --network private \
  --security-group ssh-icmp

Create VM with config drive

openstack server create test-vm-01 \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name kolla-key \
  --network private \
  --security-group ssh-icmp \
  --config-drive true

Purpose:
Useful where metadata service access is unreliable.

Create VM with cloud-init user data

openstack server create web-01 \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name kolla-key \
  --network private \
  --security-group ssh-icmp \
  --user-data cloud-init.yaml

Create VM on specific availability zone

openstack server create test-vm-az \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name kolla-key \
  --network private \
  --availability-zone nova

Create VM on a specific compute host

Admin-only:

openstack server create pinned-vm \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name kolla-key \
  --network private \
  --availability-zone nova:cmp

Purpose:
Forces scheduling to a specific compute host, for example cmp or gpu.

Create GPU test VM

Based on your environment:

openstack server create gpu-test-01 \
  --image ubuntu-24.04 \
  --flavor gpu.1x \
  --key-name kolla-key \
  --network private \
  --security-group ssh-icmp \
  --availability-zone nova:gpu

Reboot server

Soft reboot:

openstack server reboot gpu-test-01

Hard reboot:

openstack server reboot --hard gpu-test-01

Stop / start server

openstack server stop gpu-test-01
openstack server start gpu-test-01

Pause / unpause

openstack server pause gpu-test-01
openstack server unpause gpu-test-01

Suspend / resume

openstack server suspend gpu-test-01
openstack server resume gpu-test-01

Shelve / unshelve

openstack server shelve gpu-test-01
openstack server unshelve gpu-test-01

Purpose:
Useful for freeing compute resources while preserving instance state.

Resize server

openstack server resize \
  --flavor m1.medium \
  test-vm-01

Confirm resize:

openstack server resize confirm test-vm-01

Revert resize:

openstack server resize revert test-vm-01

Rebuild server

openstack server rebuild \
  --image ubuntu-24.04 \
  test-vm-01

Purpose:
Reinstalls the VM from image while keeping the server identity.

Delete server

openstack server delete test-vm-01

Wait for deletion:

openstack server delete test-vm-01 --wait

---

14. Server diagnostics and console commands

Show console log

openstack console log show gpu-test-01

Limit output:

openstack console log show gpu-test-01 --lines 100

Purpose:
Essential for debugging cloud-init, boot problems, DHCP, metadata issues and kernel errors.

Get VNC/SPICE console URL

openstack console url show gpu-test-01

Show server events

openstack server event list gpu-test-01

Show event detail

openstack server event show gpu-test-01 <request-id>

Purpose:
Useful for debugging failed builds, scheduling problems, spawn errors and quota issues.

---

15. Hypervisor / compute host commands

Admin-only.

List hypervisors

openstack hypervisor list

Show hypervisor details

openstack hypervisor show cmp
openstack hypervisor show gpu

Show hypervisor stats

openstack hypervisor stats show

Purpose:
Shows total vCPUs, memory, disk and usage across compute nodes.

List compute services

openstack compute service list

Filter Nova compute:

openstack compute service list --service nova-compute

Disable compute host

openstack compute service set \
  --disable \
  --disable-reason "maintenance" \
  gpu nova-compute

Enable compute host

openstack compute service set \
  --enable \
  gpu nova-compute

Purpose:
Drains or returns a compute node to scheduling.

---

16. Availability zone commands

List availability zones

openstack availability zone list

List compute availability zones

openstack availability zone list --compute

List network availability zones

openstack availability zone list --network

---

17. Placement commands

Placement tracks resource providers, inventories, traits and allocations.

List resource providers

openstack resource provider list

Show resource provider

openstack resource provider show <resource-provider-id>

List inventories

openstack resource provider inventory list <resource-provider-id>

List traits

openstack resource provider trait list <resource-provider-id>

List allocations for a server

First get server UUID:

openstack server show gpu-test-01 -c id -f value

Then:

openstack resource provider allocation show <server-uuid>

Purpose:
Critical for debugging why a server consumed vCPU, RAM, disk, PCI, vGPU or custom resources.

List allocation candidates

Example:

openstack allocation candidate list \
  --resource VCPU=4 \
  --resource MEMORY_MB=8192 \
  --resource DISK_GB=80

Purpose:
Shows whether Placement can find providers satisfying a resource request.

---

18. Volume / Cinder commands

The unified OpenStack CLI exposes Block Storage resources under openstack volume. The OpenStackClient quota object also spans Block Storage, Compute and Network resources.

List volumes

openstack volume list

Show volume

openstack volume show data-vol-01

Create volume

openstack volume create data-vol-01 \
  --size 20

Create volume from image

openstack volume create ubuntu-root-vol \
  --image ubuntu-24.04 \
  --size 40 \
  --bootable

Boot VM from volume

openstack server create boot-from-volume-01 \
  --flavor m1.small \
  --volume ubuntu-root-vol \
  --key-name kolla-key \
  --network private \
  --security-group ssh-icmp

Attach volume to server

openstack server add volume test-vm-01 data-vol-01

The OpenStack Cinder administration documentation describes creating a volume and attaching it to an instance through openstack volume create and openstack server add volume.

Detach volume

openstack server remove volume test-vm-01 data-vol-01

Extend volume

openstack volume set data-vol-01 --size 50

Purpose:
Increases volume size. The guest filesystem still needs to be expanded inside the VM.

Create snapshot

openstack volume snapshot create \
  --volume data-vol-01 \
  data-vol-01-snap

List snapshots

openstack volume snapshot list

Create volume from snapshot

openstack volume create restored-vol \
  --snapshot data-vol-01-snap \
  --size 20

Delete volume

openstack volume delete data-vol-01

Force delete volume

openstack volume delete --force data-vol-01

Use with care.

---

19. Volume type commands

List volume types

openstack volume type list

Create volume type

openstack volume type create fast-ssd

Set backend extra spec

openstack volume type set fast-ssd \
  --property volume_backend_name=ssd-backend

Create volume using type

openstack volume create db-vol-01 \
  --size 100 \
  --type fast-ssd

---

20. Quota commands

OpenStackClient exposes quotas across Compute, Block Storage and Network resources as a single object.

Show project quota

openstack quota show lab-project

Show detailed quota

openstack quota show lab-project --detail

Set compute quotas

openstack quota set lab-project \
  --cores 32 \
  --ram 65536 \
  --instances 20

Set volume quotas

openstack quota set lab-project \
  --volumes 20 \
  --gigabytes 1000 \
  --snapshots 20

Set network quotas

openstack quota set lab-project \
  --networks 10 \
  --subnets 10 \
  --routers 5 \
  --floating-ips 20 \
  --ports 100 \
  --security-groups 20

Reset quota to defaults

openstack quota delete lab-project

---

21. Object Storage / Swift commands

Only relevant if Swift is enabled.

List containers

openstack container list

Create container

openstack container create backups

Upload object

openstack object create backups ./file.tar.gz

List objects

openstack object list backups

Download object

openstack object save backups file.tar.gz

Delete object

openstack object delete backups file.tar.gz

---

22. Network troubleshooting commands

These are useful when debugging VM connectivity.

Show VM IPs

openstack server show gpu-test-01 -c addresses

Show VM ports

openstack port list --server gpu-test-01

Show DHCP agent hosting network

openstack network agent list \
  --agent-type dhcp \
  --network private

List all network agents

openstack network agent list

Show routers

openstack router list

Show router interfaces

openstack port list --router lab-router

Show floating IP mappings

openstack floating ip list

Show security group rules

openstack security group rule list

Find the Neutron port for a fixed IP

openstack port list --fixed-ip ip-address=10.10.10.36

Show port binding host

openstack port show <port-id> \
  -c binding_host_id \
  -c binding_vif_type \
  -c binding_vif_details \
  -c fixed_ips \
  -c status

Purpose:
Shows where Neutron thinks the port is bound and whether the backend binding succeeded.

---

23. DHCP namespace access pattern

In your earlier GPU VM case, you used a command like:

sudo ip netns exec qdhcp-54829687-5a62-4d95-a7d0-42f3e30f7dbf \
  ssh -i /home/sont/.ssh/id_ed25519_kolla ubuntu@10.10.10.36

To find the namespace:

sudo ip netns list

DHCP namespaces usually look like:

qdhcp-<network-uuid>

To map network UUID:

openstack network show private -c id -f value

Then:

sudo ip netns exec qdhcp-<network-id> ip addr
sudo ip netns exec qdhcp-<network-id> ip route
sudo ip netns exec qdhcp-<network-id> ping 10.10.10.36

Purpose:
This tests VM connectivity from inside the Neutron DHCP namespace, which is often closer to the tenant network than the host shell.

---

24. Metadata service troubleshooting

From inside VM

curl http://169.254.169.254/openstack/latest/meta_data.json

From network node / controller

List metadata-related namespaces:

sudo ip netns list

Check router namespace:

sudo ip netns exec qrouter-<router-id> ip route

Check DHCP namespace:

sudo ip netns exec qdhcp-<network-id> ip addr

Purpose:
Used when cloud-init fails to retrieve SSH keys, hostname, user-data, or metadata.

---

25. Server build failure diagnostics

Show failed servers

openstack server list --all-projects --status ERROR

Show fault field

openstack server show <server-id> -c fault

Show event list

openstack server event list <server-id>

Show detailed build event

openstack server event show <server-id> <request-id>

Common causes:

No valid host found
Quota exceeded
Image not found
Flavor not found
Network unavailable
Port binding failed
Insufficient compute resources
PCI/GPU passthrough allocation failed

---

26. JSON / YAML output for automation

JSON output

openstack server list -f json

YAML output

openstack server show gpu-test-01 -f yaml

Single value output

openstack server show gpu-test-01 -c id -f value

Select columns

openstack server list \
  -c ID \
  -c Name \
  -c Status \
  -c Networks \
  -c Image \
  -c Flavor

Combine with jq

openstack server list -f json | jq -r '.[] | "\(.Name) \(.Status)"'

Get server UUID

SERVER_ID=$(openstack server show gpu-test-01 -c id -f value)
echo "$SERVER_ID"

---

27. Useful admin inventory commands

Services

openstack service list

Endpoints

openstack endpoint list

Compute services

openstack compute service list

Network agents

openstack network agent list

Hypervisors

openstack hypervisor list

Images

openstack image list

Flavors

openstack flavor list

Projects

openstack project list

Servers across all projects

openstack server list --all-projects --long

---

28. Kolla-Ansible aligned checks

These are not pure openstack commands, but they are essential in your deployment.

Check containers

docker ps --format 'table {{.Names}}\t{{.Status}}'

Check Nova containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep nova

Check Neutron containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep neutron

Check Keystone

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep keystone

Check Horizon

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep horizon

Run OpenStack CLI inside toolbox

docker exec -it kolla_toolbox bash
source /etc/kolla/admin-openrc.sh
openstack service list

Depending on your Kolla deployment, the admin RC file may be mounted differently, so your normal host-side method remains:

source /etc/kolla/admin-openrc.sh

---

29. Practical VM creation workflow

A clean end-to-end workflow:

source /etc/kolla/admin-openrc.sh

openstack project create lab-project
openstack user create labuser --password 'StrongPasswordHere' --project lab-project
openstack role add --project lab-project --user labuser member

openstack network create lab-net

openstack subnet create lab-subnet \
  --network lab-net \
  --subnet-range 10.20.20.0/24 \
  --gateway 10.20.20.1 \
  --dns-nameserver 1.1.1.1

openstack router create lab-router
openstack router add subnet lab-router lab-subnet
openstack router set lab-router --external-gateway public

openstack security group create lab-ssh-icmp
openstack security group rule create lab-ssh-icmp --protocol icmp
openstack security group rule create lab-ssh-icmp --protocol tcp --dst-port 22

openstack keypair create \
  --public-key ~/.ssh/id_ed25519_kolla.pub \
  lab-key

openstack server create lab-vm-01 \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name lab-key \
  --network lab-net \
  --security-group lab-ssh-icmp

Then check:

openstack server list
openstack server show lab-vm-01
openstack console log show lab-vm-01 --lines 100

---

30. Practical GPU VM workflow

source /etc/kolla/admin-openrc.sh

openstack flavor create gpu.1x \
  --vcpus 4 \
  --ram 8192 \
  --disk 80

openstack flavor set gpu.1x \
  --property pci_passthrough:alias='gpu:1'

openstack server create gpu-test-01 \
  --image ubuntu-24.04 \
  --flavor gpu.1x \
  --key-name kolla-key \
  --network private \
  --security-group ssh-icmp \
  --availability-zone nova:gpu

Check status:

openstack server show gpu-test-01 \
  -c status \
  -c addresses \
  -c OS-EXT-SRV-ATTR:host \
  -c fault

Check events:

openstack server event list gpu-test-01

SSH through the DHCP namespace if required:

sudo ip netns list

sudo ip netns exec qdhcp-<network-id> \
  ssh -i /home/sont/.ssh/id_ed25519_kolla ubuntu@10.10.10.36

Inside the VM:

lspci | grep -i nvidia
nvidia-smi

---

31. High-value troubleshooting one-liners

Find where a VM is running

openstack server show gpu-test-01 \
  -c OS-EXT-SRV-ATTR:host \
  -c OS-EXT-AZ:availability_zone \
  -c status \
  -c addresses

Find failed builds

openstack server list --all-projects --status ERROR

Show failure reason

openstack server show <server> -c fault

Check Nova compute health

openstack compute service list --service nova-compute

Check Neutron agent health

openstack network agent list

Check resource pressure

openstack hypervisor stats show

Check project quota

openstack quota show <project> --detail

Check API endpoints

openstack endpoint list

Check available images and flavors

openstack image list
openstack flavor list

Check server console

openstack console log show <server> --lines 200

---

32. Advanced command patterns

Use variables safely

PROJECT=lab-project
IMAGE=ubuntu-24.04
FLAVOR=m1.small
NETWORK=private
KEY=kolla-key
SECGRP=ssh-icmp
SERVER=lab-vm-01

openstack server create "$SERVER" \
  --image "$IMAGE" \
  --flavor "$FLAVOR" \
  --key-name "$KEY" \
  --network "$NETWORK" \
  --security-group "$SECGRP"

Wait for server creation

openstack server create lab-vm-01 \
  --image ubuntu-24.04 \
  --flavor m1.small \
  --key-name kolla-key \
  --network private \
  --wait

Wait for delete

openstack server delete lab-vm-01 --wait

Get first fixed IP

openstack server show lab-vm-01 -f json | jq -r '.addresses'

List servers by host

openstack server list --all-projects --host gpu

List servers by project

openstack server list --project lab-project

List only server names

openstack server list -f value -c Name

---

33. Commands I would use most in your homelab

For your current topology, these are the highest-value daily commands:

source /etc/kolla/admin-openrc.sh

openstack service list

openstack endpoint list

openstack compute service list

openstack network agent list

openstack hypervisor list

openstack hypervisor stats show

openstack server list --all-projects --long

openstack server show gpu-test-01

openstack server event list gpu-test-01

openstack console log show gpu-test-01 --lines 200

openstack port list --server gpu-test-01

openstack port show <port-id>

openstack floating ip list

openstack router list

openstack security group rule list

openstack quota show admin --detail

openstack resource provider list

openstack resource provider allocation show <server-uuid>

These give you coverage across Keystone, Nova, Neutron, Glance, Placement and Cinder and are the core operator commands for debugging OpenStack in a Kolla-Ansible homelab.

Below is an operator-focused list of basic and advanced Docker commands for OpenStack operations and troubleshooting, especially relevant to Kolla-Ansible, where OpenStack services run as Docker containers.

I will assume your usual pattern is something like:

source /opt/kolla-venv/bin/activate
source /etc/kolla/admin-openrc.sh

and that your OpenStack services are containerised as:

keystone
nova_api
nova_compute
neutron_server
neutron_l3_agent
neutron_dhcp_agent
glance_api
placement_api
horizon
mariadb
rabbitmq
memcached
haproxy
keepalived
kolla_toolbox

---

1. Basic Docker inspection commands

List running OpenStack containers

docker ps

More readable:

docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'

Purpose:
Shows which OpenStack containers are currently running.

Useful in Kolla:

docker ps --format 'table {{.Names}}\t{{.Status}}' | sort

---

List all containers, including stopped ones

docker ps -a

Purpose:
Useful when a service container is failing and restarting or has exited.

Example:

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' | sort

---

Filter OpenStack service containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep nova

Examples:

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep neutron
docker ps --format 'table {{.Names}}\t{{.Status}}' | grep keystone
docker ps --format 'table {{.Names}}\t{{.Status}}' | grep glance
docker ps --format 'table {{.Names}}\t{{.Status}}' | grep placement
docker ps --format 'table {{.Names}}\t{{.Status}}' | grep horizon
docker ps --format 'table {{.Names}}\t{{.Status}}' | grep haproxy

Purpose:
Quickly checks whether a service group is healthy.

---

2. Container status and health

Check container health status

docker inspect --format='{{.State.Status}} {{.State.Health.Status}}' <container>

Example:

docker inspect --format='{{.State.Status}} {{.State.Health.Status}}' nova_api

Some containers may not have Docker health checks configured. In that case, use:

docker inspect --format='{{.State.Status}}' nova_api

---

Show restart count

docker inspect --format='{{.RestartCount}}' <container>

Example:

docker inspect --format='{{.RestartCount}}' neutron_server

Purpose:
High restart count usually means the service has been crashing repeatedly.

---

Show container start time

docker inspect --format='{{.State.StartedAt}}' <container>

Example:

docker inspect --format='{{.State.StartedAt}}' nova_compute

---

Show exit code of stopped container

docker inspect --format='{{.State.ExitCode}}' <container>

Example:

docker inspect --format='{{.State.ExitCode}}' horizon

---

3. Logs: most important troubleshooting commands

Show logs for a container

docker logs <container>

Example:

docker logs nova_api

---

Follow logs live

docker logs -f <container>

Example:

docker logs -f neutron_server

---

Show recent logs only

docker logs --tail 100 <container>

Example:

docker logs --tail 100 nova_compute

---

Follow recent logs

docker logs -f --tail 100 <container>

Example:

docker logs -f --tail 100 neutron_l3_agent

---

Show logs with timestamps

docker logs -t --tail 200 <container>

Example:

docker logs -t --tail 200 keystone

---

Search logs for errors

docker logs <container> 2>&1 | grep -i error

Examples:

docker logs nova_compute 2>&1 | grep -i error
docker logs neutron_server 2>&1 | grep -i error
docker logs glance_api 2>&1 | grep -i error
docker logs placement_api 2>&1 | grep -i error

---

Search for tracebacks

docker logs <container> 2>&1 | grep -i traceback

Example:

docker logs nova_api 2>&1 | grep -i traceback

---

Search for common failure keywords

docker logs <container> 2>&1 | grep -Ei 'error|failed|traceback|exception|denied|refused|timeout|unreachable'

Example:

docker logs neutron_server 2>&1 | grep -Ei 'error|failed|traceback|exception|denied|refused|timeout|unreachable'

---

4. Entering OpenStack containers

Open an interactive shell in a container

docker exec -it <container> bash

Example:

docker exec -it nova_api bash

Some containers may not have bash; use sh:

docker exec -it <container> sh

---

Run one command inside a container

docker exec <container> <command>

Examples:

docker exec nova_api hostname
docker exec neutron_server hostname
docker exec keystone hostname

---

Run OpenStack CLI from kolla_toolbox

docker exec -it kolla_toolbox bash

Then inside the container:

source /etc/kolla/admin-openrc.sh
openstack service list
openstack endpoint list
openstack compute service list
openstack network agent list

Or one-liner:

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack service list'

Purpose:
kolla_toolbox is often the best container for running OpenStack administrative commands because it contains the required CLI tooling.

---

5. Restarting OpenStack containers

Restart one container

docker restart <container>

Example:

docker restart neutron_server

---

Restart Nova API

docker restart nova_api

---

Restart Nova compute

docker restart nova_compute

Use case:
After Nova config changes on a compute node, or when Nova compute has lost connectivity to RabbitMQ, Placement, libvirt, or Neutron.

---

Restart Neutron services

docker restart neutron_server
docker restart neutron_openvswitch_agent
docker restart neutron_l3_agent
docker restart neutron_dhcp_agent
docker restart neutron_metadata_agent

Use case:
When networking, DHCP, metadata, router namespaces, or port binding are not working correctly.

---

Restart Horizon

docker restart horizon

Use case:
When the Horizon web dashboard is not responding, showing stale config, or failing behind HAProxy.

---

Restart HAProxy

docker restart haproxy

Use case:
When API VIP routing or Horizon routing is broken.

In Kolla-Ansible, prefer reconfiguration for persistent changes:

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" reconfigure --tags haproxy,horizon

---

6. Inspecting container configuration

Full inspect

docker inspect <container>

Example:

docker inspect nova_api

---

Show mounted volumes

docker inspect --format='{{json .Mounts}}' <container> | jq

Example:

docker inspect --format='{{json .Mounts}}' nova_compute | jq

Purpose:
Shows which host paths are mounted into the container.

Useful for config/log troubleshooting.

---

Show container environment variables

docker inspect --format='{{json .Config.Env}}' <container> | jq

Example:

docker inspect --format='{{json .Config.Env}}' keystone | jq

---

Show container image

docker inspect --format='{{.Config.Image}}' <container>

Example:

docker inspect --format='{{.Config.Image}}' nova_api

---

Show container network mode

docker inspect --format='{{.HostConfig.NetworkMode}}' <container>

Example:

docker inspect --format='{{.HostConfig.NetworkMode}}' neutron_server

Kolla containers commonly use host networking.

---

7. Checking OpenStack config files inside containers

Nova config

docker exec nova_api grep -v '^#' /etc/nova/nova.conf | grep -v '^$'

On compute node:

docker exec nova_compute grep -v '^#' /etc/nova/nova.conf | grep -v '^$'

---

Neutron config

docker exec neutron_server grep -v '^#' /etc/neutron/neutron.conf | grep -v '^$'

Open vSwitch agent config:

docker exec neutron_openvswitch_agent grep -v '^#' /etc/neutron/plugins/ml2/openvswitch_agent.ini | grep -v '^$'

ML2 plugin config:

docker exec neutron_server grep -v '^#' /etc/neutron/plugins/ml2/ml2_conf.ini | grep -v '^$'

---

Keystone config

docker exec keystone grep -v '^#' /etc/keystone/keystone.conf | grep -v '^$'

---

Glance config

docker exec glance_api grep -v '^#' /etc/glance/glance-api.conf | grep -v '^$'

---

Placement config

docker exec placement_api grep -v '^#' /etc/placement/placement.conf | grep -v '^$'

---

Horizon config

docker exec horizon grep -v '^#' /etc/openstack-dashboard/local_settings | grep -v '^$'

or:

docker exec horizon grep -Ei 'OPENSTACK|ALLOWED_HOSTS|WEBROOT|SESSION|CACHE|KEYSTONE' /etc/openstack-dashboard/local_settings

---

8. Checking ports and listeners inside containers

Check listening ports inside a container

docker exec <container> ss -tlnp

Examples:

docker exec keystone ss -tlnp
docker exec nova_api ss -tlnp
docker exec neutron_server ss -tlnp
docker exec glance_api ss -tlnp
docker exec placement_api ss -tlnp
docker exec horizon ss -tlnp
docker exec haproxy ss -tlnp

---

Check HAProxy listeners

docker exec haproxy ss -tlnp

Then inspect HAProxy config:

docker exec haproxy grep -n 'listen\|frontend\|backend\|server' /etc/haproxy/haproxy.cfg

---

Check Horizon listener

docker exec horizon ss -tlnp

In your earlier Horizon case, this kind of command helped show where Horizon was actually listening.

---

9. API endpoint testing from containers

Test Keystone from kolla_toolbox

docker exec kolla_toolbox bash -lc 'curl -s -i http://127.0.0.1:5000/v3'

or against VIP:

docker exec kolla_toolbox bash -lc 'curl -s -i http://192.168.1.50:5000/v3'

---

Test Horizon

docker exec kolla_toolbox bash -lc 'curl -s -i http://192.168.1.51/'

or from the host:

curl -i http://192.168.1.51/

---

Test Nova API

docker exec kolla_toolbox bash -lc 'curl -s -i http://192.168.1.50:8774'

---

Test Placement API

docker exec kolla_toolbox bash -lc 'curl -s -i http://192.168.1.50:8778'

---

Test Glance API

docker exec kolla_toolbox bash -lc 'curl -s -i http://192.168.1.50:9292'

---

Test Neutron API

docker exec kolla_toolbox bash -lc 'curl -s -i http://192.168.1.50:9696'

---

10. Database troubleshooting with Docker

Kolla commonly runs MariaDB as a container.

Check MariaDB container

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep mariadb

---

View MariaDB logs

docker logs --tail 200 mariadb

---

Enter MariaDB container

docker exec -it mariadb bash

---

Open MariaDB shell

Depending on your deployment:

docker exec -it mariadb mysql -uroot -p

or using Kolla’s config/password files:

docker exec -it mariadb bash

Then inspect:

ls -l /etc/kolla/
ls -l /var/lib/mysql/

---

Check databases

docker exec -it mariadb mysql -uroot -p -e "SHOW DATABASES;"

Expected OpenStack databases may include:

keystone
nova
nova_api
nova_cell0
neutron
glance
placement
cinder

---

Check database connectivity from service container

Example from Nova:

docker exec nova_api bash -lc 'grep -i connection /etc/nova/nova.conf'

Example from Neutron:

docker exec neutron_server bash -lc 'grep -i connection /etc/neutron/neutron.conf'

---

11. RabbitMQ troubleshooting with Docker

RabbitMQ is critical for Nova, Neutron, Cinder and other services.

Check RabbitMQ container

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep rabbitmq

---

View RabbitMQ logs

docker logs --tail 200 rabbitmq

---

Check RabbitMQ status

docker exec rabbitmq rabbitmqctl status

---

List RabbitMQ queues

docker exec rabbitmq rabbitmqctl list_queues

More useful:

docker exec rabbitmq rabbitmqctl list_queues name messages consumers state

---

List RabbitMQ connections

docker exec rabbitmq rabbitmqctl list_connections name user peer_host state

---

List RabbitMQ exchanges

docker exec rabbitmq rabbitmqctl list_exchanges

---

List RabbitMQ users

docker exec rabbitmq rabbitmqctl list_users

---

Check for stuck queues

docker exec rabbitmq rabbitmqctl list_queues name messages consumers state | sort -k2 -nr | head

Purpose:
If queues are growing and consumers are zero, an OpenStack service may be down or disconnected.

---

12. Memcached troubleshooting

Keystone and other services often rely on memcached.

Check memcached container

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep memcached

---

Check memcached listener

docker exec memcached ss -tlnp

---

View logs

docker logs --tail 100 memcached

---

13. Libvirt / Nova compute troubleshooting

In Kolla, libvirt usually runs in a container named something like:

nova_libvirt

Check libvirt container

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep libvirt

---

List libvirt domains

docker exec nova_libvirt virsh list --all

Purpose:
Shows instances known to libvirt on the compute host.

---

Show domain XML

docker exec nova_libvirt virsh dumpxml <domain-name>

Example:

docker exec nova_libvirt virsh dumpxml instance-00000001

---

Check GPU passthrough in XML

docker exec nova_libvirt virsh dumpxml <domain-name> | grep -i hostdev -A20

or:

docker exec nova_libvirt virsh dumpxml <domain-name> | grep -Ei 'hostdev|pci|vendor|product|address'

Purpose:
Confirms whether the GPU PCI device was passed into the instance.

---

Check libvirt node devices

docker exec nova_libvirt virsh nodedev-list

Filter PCI:

docker exec nova_libvirt virsh nodedev-list | grep pci

---

Show PCI node device details

docker exec nova_libvirt virsh nodedev-dumpxml pci_0000_00_06_0

---

Check QEMU processes

ps aux | grep qemu

or inside relevant container/host context:

docker exec nova_libvirt ps aux | grep qemu

---

Check Nova compute logs for spawn errors

docker logs nova_compute 2>&1 | grep -Ei 'spawn|libvirt|qemu|pci|vfio|error|traceback|NoValidHost'

---

14. Neutron namespace troubleshooting

These commands are host-side, but closely related to Dockerised Neutron.

List network namespaces

sudo ip netns list

Typical namespaces:

qdhcp-<network-id>
qrouter-<router-id>

---

Map OpenStack network to namespace

openstack network show private -c id -f value

Then:

sudo ip netns exec qdhcp-<network-id> ip addr

---

Test VM reachability from DHCP namespace

sudo ip netns exec qdhcp-<network-id> ping 10.10.10.36

---

SSH to VM through DHCP namespace

sudo ip netns exec qdhcp-<network-id> \
  ssh -i /home/sont/.ssh/id_ed25519_kolla ubuntu@10.10.10.36

This is the same style of access pattern you used for gpu-test-01.

---

Inspect router namespace

sudo ip netns exec qrouter-<router-id> ip addr
sudo ip netns exec qrouter-<router-id> ip route
sudo ip netns exec qrouter-<router-id> iptables -t nat -S

---

Check DHCP namespace

sudo ip netns exec qdhcp-<network-id> ip addr
sudo ip netns exec qdhcp-<network-id> ip route
sudo ip netns exec qdhcp-<network-id> ps aux

---

15. Open vSwitch troubleshooting

These are often host-side, but relevant to Neutron containers.

Check OVS containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep openvswitch

Common names:

openvswitch_db
openvswitch_vswitchd
neutron_openvswitch_agent

---

Show OVS bridges

docker exec openvswitch_vswitchd ovs-vsctl show

If ovs-vsctl is not in that container, run from host:

sudo ovs-vsctl show

---

List OVS bridges

docker exec openvswitch_vswitchd ovs-vsctl list-br

---

List ports on a bridge

docker exec openvswitch_vswitchd ovs-vsctl list-ports br-int
docker exec openvswitch_vswitchd ovs-vsctl list-ports br-ex

---

Show OpenFlow rules

docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-int

or from host:

sudo ovs-ofctl dump-flows br-int

---

Check Neutron OVS agent logs

docker logs neutron_openvswitch_agent 2>&1 | grep -Ei 'error|failed|bridge|port|ovs|binding'

---

16. HAProxy / Keepalived troubleshooting

Check HAProxy

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep haproxy

---

Show HAProxy logs

docker logs --tail 200 haproxy

---

Check HAProxy listeners

docker exec haproxy ss -tlnp

---

Inspect HAProxy config

docker exec haproxy grep -n 'listen\|frontend\|backend\|bind\|server' /etc/haproxy/haproxy.cfg

---

Validate HAProxy config

docker exec haproxy haproxy -c -f /etc/haproxy/haproxy.cfg

---

Check Keepalived

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep keepalived

---

Show Keepalived logs

docker logs --tail 200 keepalived

---

Check VIP on host

ip addr | grep 192.168.1.50
ip addr | grep 192.168.1.51

Adjust the IPs to match your OpenStack internal/public VIPs.

---

17. Horizon troubleshooting

Check Horizon container

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep horizon

---

Show Horizon logs

docker logs --tail 200 horizon

---

Follow Horizon logs

docker logs -f --tail 100 horizon

---

Check Horizon listener

docker exec horizon ss -tlnp

---

Test Horizon locally

curl -i http://192.168.1.51/

or from toolbox:

docker exec kolla_toolbox bash -lc 'curl -i http://192.168.1.51/'

---

Inspect Horizon local settings

docker exec horizon grep -Ei 'OPENSTACK|ALLOWED_HOSTS|WEBROOT|SESSION|CACHE|KEYSTONE' /etc/openstack-dashboard/local_settings

---

18. Glance troubleshooting

Check Glance

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep glance

---

Glance logs

docker logs --tail 200 glance_api

---

Search Glance logs

docker logs glance_api 2>&1 | grep -Ei 'error|failed|traceback|store|filesystem|ceph|permission'

---

Inspect Glance config

docker exec glance_api grep -v '^#' /etc/glance/glance-api.conf | grep -v '^$'

---

Check Glance API listener

docker exec glance_api ss -tlnp

---

19. Nova troubleshooting

Check Nova containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep nova

---

Common Nova containers

nova_api
nova_conductor
nova_scheduler
nova_compute
nova_libvirt
nova_metadata
nova_novncproxy

---

Check Nova API logs

docker logs --tail 200 nova_api

---

Check Nova scheduler logs

docker logs --tail 200 nova_scheduler

---

Check Nova compute logs

docker logs --tail 200 nova_compute

---

Search for scheduling problems

docker logs nova_scheduler 2>&1 | grep -Ei 'NoValidHost|Filter|Placement|Allocation|error|failed'

---

Search for compute spawn problems

docker logs nova_compute 2>&1 | grep -Ei 'spawn|libvirt|qemu|NoValidHost|BuildAbort|error|failed|traceback'

---

Search for Placement problems

docker logs nova_compute 2>&1 | grep -Ei 'placement|allocation|resource provider|inventory|trait'

---

Check Nova services from toolbox

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack compute service list'

---

Check hypervisors from toolbox

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack hypervisor list'

---

20. Neutron troubleshooting

Check Neutron containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep neutron

---

Common Neutron containers

neutron_server
neutron_openvswitch_agent
neutron_l3_agent
neutron_dhcp_agent
neutron_metadata_agent

---

Check Neutron server logs

docker logs --tail 200 neutron_server

---

Check L3 agent logs

docker logs --tail 200 neutron_l3_agent

---

Check DHCP agent logs

docker logs --tail 200 neutron_dhcp_agent

---

Check metadata agent logs

docker logs --tail 200 neutron_metadata_agent

---

Search Neutron errors

docker logs neutron_server 2>&1 | grep -Ei 'error|failed|traceback|binding|port|agent|ml2'

---

Search DHCP problems

docker logs neutron_dhcp_agent 2>&1 | grep -Ei 'error|failed|dnsmasq|dhcp|namespace'

---

Search router/L3 problems

docker logs neutron_l3_agent 2>&1 | grep -Ei 'error|failed|router|namespace|iptables|snat|floating'

---

Check Neutron agents from toolbox

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack network agent list'

---

21. Keystone troubleshooting

Check Keystone

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep keystone

---

Keystone logs

docker logs --tail 200 keystone

---

Search Keystone errors

docker logs keystone 2>&1 | grep -Ei 'error|failed|traceback|token|fernet|credential|assignment|ldap'

---

Check Keystone listener

docker exec keystone ss -tlnp

---

Test token issue

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack token issue'

---

Check service catalog

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack catalog list'

---

22. Placement troubleshooting

Check Placement

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep placement

---

Placement logs

docker logs --tail 200 placement_api

---

Search Placement errors

docker logs placement_api 2>&1 | grep -Ei 'error|failed|traceback|allocation|resource|inventory|trait'

---

Placement API test

docker exec kolla_toolbox bash -lc 'curl -i http://192.168.1.50:8778'

---

List resource providers

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack resource provider list'

---

23. Cinder troubleshooting

Only relevant if Cinder is enabled.

Check Cinder containers

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep cinder

Common containers:

cinder_api
cinder_scheduler
cinder_volume
cinder_backup

---

Cinder API logs

docker logs --tail 200 cinder_api

---

Cinder scheduler logs

docker logs --tail 200 cinder_scheduler

---

Cinder volume logs

docker logs --tail 200 cinder_volume

---

Search Cinder errors

docker logs cinder_volume 2>&1 | grep -Ei 'error|failed|traceback|backend|iscsi|ceph|rbd|volume'

---

Check Cinder services

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack volume service list'

---

24. Docker image operations

List Kolla images

docker images | grep kolla

---

Show image sizes

docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}' | sort

---

Check image used by a container

docker inspect --format='{{.Config.Image}}' nova_api

---

Pull updated images manually

Usually use Kolla-Ansible:

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" pull

Docker-level pull example:

docker pull <image-name>:<tag>

Use Kolla-Ansible rather than manual pulls for consistency.

---

25. Docker volume and mount troubleshooting

List Docker volumes

docker volume ls

---

Inspect a Docker volume

docker volume inspect <volume>

---

Show container mounts

docker inspect --format='{{json .Mounts}}' <container> | jq

Example:

docker inspect --format='{{json .Mounts}}' mariadb | jq

---

Check mounted config directory

Kolla commonly stores generated config under:

/etc/kolla/

Examples:

ls -la /etc/kolla/
ls -la /etc/kolla/nova-api/
ls -la /etc/kolla/neutron-server/
ls -la /etc/kolla/horizon/

---

Compare host config and container config

Example for Nova API:

ls -la /etc/kolla/nova-api/
docker exec nova_api ls -la /etc/nova/

Example for Neutron:

ls -la /etc/kolla/neutron-server/
docker exec neutron_server ls -la /etc/neutron/

---

26. Docker resource usage

Show live resource usage

docker stats

---

Show selected containers

docker stats nova_api neutron_server nova_compute rabbitmq mariadb

---

One-shot stats

docker stats --no-stream

More readable:

docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}'

Purpose:
Useful for finding overloaded services such as RabbitMQ, MariaDB, Nova, Neutron, or Horizon.

---

27. Docker networking checks

List Docker networks

docker network ls

In Kolla, many containers use host networking, so this may be less interesting than in normal Docker deployments.

---

Inspect container IP/network settings

docker inspect --format='{{json .NetworkSettings}}' <container> | jq

Example:

docker inspect --format='{{json .NetworkSettings}}' nova_api | jq

---

Check whether container uses host networking

docker inspect --format='{{.HostConfig.NetworkMode}}' <container>

Example:

docker inspect --format='{{.HostConfig.NetworkMode}}' neutron_server

---

28. Docker process inspection

Show processes inside a container

docker top <container>

Examples:

docker top nova_api
docker top neutron_server
docker top rabbitmq
docker top mariadb

---

Run ps inside container

docker exec <container> ps aux

Example:

docker exec nova_api ps aux

---

29. Copying files in and out of containers

Copy file from container to host

docker cp <container>:/path/in/container /path/on/host

Example:

docker cp nova_api:/etc/nova/nova.conf ./nova.conf

---

Copy file from host to container

docker cp ./file.conf <container>:/tmp/file.conf

Use with care. In Kolla, persistent config should be managed through /etc/kolla/ and Kolla-Ansible reconfiguration, not manual edits inside containers.

---

30. Safe cleanup commands

Remove stopped containers

docker container prune

Use caution.

---

Remove unused images

docker image prune

More aggressive:

docker image prune -a

Use caution; do not remove images you may need for rollback.

---

Show Docker disk usage

docker system df

Verbose:

docker system df -v

---

Clean unused Docker resources

docker system prune

Aggressive cleanup:

docker system prune -a

Be careful on OpenStack nodes. Do not run aggressive prune blindly on production or important lab environments.

---

31. Kolla-Ansible plus Docker operational commands

These are the commands you will probably use most often in your deployment.

Check all OpenStack containers

docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' | sort

---

Check failed or exited containers

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' | grep -Ei 'Exited|Restarting|Dead'

---

Restart one failed service

docker restart <container>

---

Prefer Kolla reconfigure for persistent changes

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" reconfigure --tags <service>

Examples:

kolla-ansible -i "$KOLLA_INVENTORY" reconfigure --tags nova
kolla-ansible -i "$KOLLA_INVENTORY" reconfigure --tags neutron
kolla-ansible -i "$KOLLA_INVENTORY" reconfigure --tags horizon
kolla-ansible -i "$KOLLA_INVENTORY" reconfigure --tags haproxy,horizon

---

Pull images

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" pull

---

Deploy

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" deploy

---

Reconfigure

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" reconfigure

---

Stop OpenStack services

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" stop

---

Destroy deployment

source /opt/kolla-venv/bin/activate
kolla-ansible -i "$KOLLA_INVENTORY" destroy

Use with extreme care.

---

32. Practical troubleshooting workflows

Workflow A: Horizon not loading

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep -Ei 'horizon|haproxy|keystone'

docker logs --tail 100 horizon
docker logs --tail 100 haproxy
docker logs --tail 100 keystone

docker exec horizon ss -tlnp
docker exec haproxy ss -tlnp

curl -i http://192.168.1.51/

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack token issue'

Expected checks:

horizon container is running
haproxy is listening on the expected VIP/IP
keystone token issue works
Horizon is reachable through the correct public URL

---

Workflow B: VM stuck in ERROR

source /etc/kolla/admin-openrc.sh

openstack server list --all-projects --status ERROR
openstack server show <vm> -c fault
openstack server event list <vm>

docker logs nova_api 2>&1 | grep -Ei 'error|traceback|failed'
docker logs nova_scheduler 2>&1 | grep -Ei 'NoValidHost|Filter|Placement|Allocation|failed|error'
docker logs nova_compute 2>&1 | grep -Ei 'spawn|libvirt|qemu|failed|error|traceback'

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack compute service list'
docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack hypervisor stats show'

Likely causes:

No valid host
Quota exceeded
Missing image
Bad flavor
Placement inventory problem
Neutron port binding failure
Libvirt/QEMU spawn failure
GPU passthrough failure

---

Workflow C: VM has no network

openstack server show <vm> -c addresses
openstack port list --server <vm>
openstack network agent list
openstack router list
openstack floating ip list
openstack security group rule list

docker logs neutron_server 2>&1 | grep -Ei 'error|failed|binding|port|ml2'
docker logs neutron_openvswitch_agent 2>&1 | grep -Ei 'error|failed|ovs|bridge|binding'
docker logs neutron_dhcp_agent 2>&1 | grep -Ei 'error|failed|dnsmasq|dhcp|namespace'
docker logs neutron_l3_agent 2>&1 | grep -Ei 'error|failed|router|namespace|iptables'

sudo ip netns list
sudo ip netns exec qdhcp-<network-id> ip addr
sudo ip netns exec qdhcp-<network-id> ping <vm-fixed-ip>

---

Workflow D: GPU passthrough not working

openstack server show gpu-test-01 -c status -c fault -c OS-EXT-SRV-ATTR:host
openstack server event list gpu-test-01

docker logs nova_scheduler 2>&1 | grep -Ei 'NoValidHost|pci|placement|allocation|trait|resource'
docker logs nova_compute 2>&1 | grep -Ei 'pci|vfio|libvirt|qemu|hostdev|error|failed'

docker exec nova_libvirt virsh list --all
docker exec nova_libvirt virsh dumpxml <instance-domain> | grep -i hostdev -A30

On the host:

lspci | grep -i nvidia
lspci -nnk -s <pci-address>
ls -l /dev/vfio/

Inside the VM:

lspci | grep -i nvidia
nvidia-smi

---

Workflow E: API endpoint not reachable

openstack endpoint list

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep -Ei 'haproxy|keystone|nova|neutron|glance|placement'

docker exec haproxy ss -tlnp
docker exec haproxy haproxy -c -f /etc/haproxy/haproxy.cfg

docker logs --tail 100 haproxy
docker logs --tail 100 keystone
docker logs --tail 100 nova_api
docker logs --tail 100 neutron_server
docker logs --tail 100 glance_api
docker logs --tail 100 placement_api

curl -i http://<vip>:5000/v3
curl -i http://<vip>:8774
curl -i http://<vip>:9696
curl -i http://<vip>:9292
curl -i http://<vip>:8778

---

33. High-value one-liners

Show unhealthy containers

docker ps -a --format 'table {{.Names}}\t{{.Status}}' | grep -Ei 'Exited|Restarting|Dead|unhealthy'

---

Show OpenStack container restart counts

for c in $(docker ps --format '{{.Names}}'); do
  echo -n "$c "
  docker inspect --format='{{.RestartCount}}' "$c"
done | sort -k2 -nr

---

Show recent errors across all running containers

for c in $(docker ps --format '{{.Names}}'); do
  echo "===== $c ====="
  docker logs --tail 100 "$c" 2>&1 | grep -Ei 'error|failed|traceback|exception|denied|refused|timeout|unreachable' || true
done

---

Show listeners for major API containers

for c in keystone nova_api neutron_server glance_api placement_api horizon haproxy; do
  echo "===== $c ====="
  docker exec "$c" ss -tlnp 2>/dev/null || true
done

---

Show all Nova-related logs with errors

for c in $(docker ps --format '{{.Names}}' | grep nova); do
  echo "===== $c ====="
  docker logs --tail 200 "$c" 2>&1 | grep -Ei 'error|failed|traceback|exception|NoValidHost|libvirt|placement' || true
done

---

Show all Neutron-related logs with errors

for c in $(docker ps --format '{{.Names}}' | grep neutron); do
  echo "===== $c ====="
  docker logs --tail 200 "$c" 2>&1 | grep -Ei 'error|failed|traceback|exception|binding|ovs|dhcp|router|namespace' || true
done

---

Run core OpenStack checks from kolla_toolbox

docker exec kolla_toolbox bash -lc '
source /etc/kolla/admin-openrc.sh
openstack service list
openstack endpoint list
openstack compute service list
openstack network agent list
openstack hypervisor list
openstack server list --all-projects --long
'

---

34. Most useful commands for your OpenStack homelab

These are the ones I would keep as your daily operator set:

docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' | sort

docker ps -a --format 'table {{.Names}}\t{{.Status}}' | grep -Ei 'Exited|Restarting|Dead|unhealthy'

docker logs -f --tail 100 nova_compute

docker logs -f --tail 100 neutron_server

docker logs -f --tail 100 neutron_openvswitch_agent

docker logs -f --tail 100 neutron_dhcp_agent

docker logs -f --tail 100 neutron_l3_agent

docker logs -f --tail 100 haproxy

docker logs -f --tail 100 horizon

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack service list'

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack compute service list'

docker exec kolla_toolbox bash -lc 'source /etc/kolla/admin-openrc.sh && openstack network agent list'

docker exec nova_libvirt virsh list --all

docker exec nova_libvirt virsh dumpxml <instance-domain> | grep -i hostdev -A30

docker exec haproxy ss -tlnp

docker exec horizon ss -tlnp

sudo ip netns list

sudo ip netns exec qdhcp-<network-id> ping <vm-fixed-ip>

These commands cover the main operational layers of a Kolla-Ansible OpenStack deployment:

Docker container state
OpenStack service health
API routing through HAProxy
Keystone authentication
Nova compute and libvirt
Neutron networking and namespaces
RabbitMQ messaging
MariaDB persistence
Horizon dashboard access
GPU passthrough verification

Kolla-Ansible Commands

Kolla-Ansible command reference for OpenStack 2025.2+, using the current documented command shape:

kolla-ansible <subcommand> -i "$KOLLA_INVENTORY"

The OpenStack 2025.2 Kolla-Ansible quickstart shows the deployment sequence as bootstrap-servers, prechecks, then deploy, each with -i <inventory> after the subcommand. It also shows post-deploy for generating OpenStack client credentials.

---

0. Standard environment setup

For your homelab, keep this pattern:

source /opt/kolla-venv/bin/activate
export KOLLA_INVENTORY=/etc/kolla/multinode

Check the tool is available:

which kolla-ansible
kolla-ansible --help
kolla-ansible deploy --help

Check installed version:

pip show kolla-ansible
pip freeze | grep -i kolla

Install the 2025.2 branch directly with pip, as shown in the 2025.2 docs:

pip install git+https://opendev.org/openstack/kolla-ansible@stable/2025.2

The 2025.2 quickstart also recommends using a Python virtual environment to avoid conflicts with system Python packages.

---

1. Dependency and initial setup commands

Install Ansible Galaxy dependencies

kolla-ansible install-deps

Purpose:
Installs the Ansible collection and role dependencies required by Kolla-Ansible.

The 2025.2 quickstart explicitly includes this as the dependency-install step before deployment configuration.

---

Generate passwords

kolla-genpwd

Purpose:
Populates /etc/kolla/passwords.yml with generated service passwords.

The 2025.2 docs state that passwords are stored in /etc/kolla/passwords.yml and can be filled by running kolla-genpwd.

---

Copy example configuration

sudo mkdir -p /etc/kolla
sudo chown "$USER:$USER" /etc/kolla

cp -r /opt/kolla-venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/
cp /opt/kolla-venv/share/kolla-ansible/ansible/inventory/multinode /etc/kolla/multinode

Purpose:
Creates the standard Kolla configuration directory and installs example globals.yml, passwords.yml, and inventory files.

---

2. Inventory validation and Ansible reachability

Basic Ansible ping

ansible -i "$KOLLA_INVENTORY" all -m ping

Purpose:
Confirms that Ansible can connect to every OpenStack target node.

---

Show inventory graph

ansible-inventory -i "$KOLLA_INVENTORY" --graph

Purpose:
Validates host grouping, for example:

control
network
compute
monitoring
storage
deployment

---

Dump inventory as JSON

ansible-inventory -i "$KOLLA_INVENTORY" --list

Purpose:
Useful when debugging variable inheritance and group membership.

---

Check host facts

ansible -i "$KOLLA_INVENTORY" all -m setup

Narrow it down:

ansible -i "$KOLLA_INVENTORY" all -m setup -a 'filter=ansible_default_ipv4'

Purpose:
Validates that Kolla-Ansible will see the expected interfaces, IP addresses, hostnames, Python interpreter and OS facts.

---

3. Core deployment lifecycle commands

Bootstrap servers

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY"

Purpose:
Prepares target hosts before OpenStack containers are deployed. This includes host-level work such as Docker engine installation/configuration, package setup, /etc/hosts, Kolla user/group creation, firewall handling, AppArmor/SELinux configuration, NTP configuration, and Kolla configuration directory preparation.

For first bootstrap where your normal Ansible user or Python interpreter is not ready yet:

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" \
  -e ansible_user=ubuntu \
  -e ansible_python_interpreter=/usr/bin/python3

The bootstrap documentation notes that the initial bootstrap environment may differ from the final Kolla environment, so ansible_user and ansible_python_interpreter can be overridden with -e.

---

Run prechecks

kolla-ansible prechecks -i "$KOLLA_INVENTORY"

Purpose:
Checks whether hosts and configuration are in a deployable state before containers are created.

The troubleshooting guide says prechecks exist to catch misconfigured environments before deployment.

---

Pull container images

kolla-ansible pull -i "$KOLLA_INVENTORY"

Purpose:
Pulls required Kolla container images to the target nodes before deployment.

This is especially useful after changing image tags, moving to a newer release, or refreshing cached images before redeploying. The troubleshooting guide notes that if release tags change, node image caches should be refreshed before a new deployment.

---

Deploy OpenStack

kolla-ansible deploy -i "$KOLLA_INVENTORY"

Purpose:
Creates and starts the OpenStack control-plane and compute service containers.

The documented deployment sequence is:

kolla-ansible bootstrap-servers -i ./all-in-one
kolla-ansible prechecks -i ./all-in-one
kolla-ansible deploy -i ./all-in-one

Replace ./all-in-one with your multinode inventory.

---

Generate post-deploy client configuration

kolla-ansible post-deploy -i "$KOLLA_INVENTORY"

Purpose:
Generates /etc/kolla/clouds.yaml for OpenStack client access.

The 2025.2 docs state that post-deploy generates /etc/kolla/clouds.yaml, which can be copied to /etc/openstack or ~/.config/openstack, or referenced via OS_CLIENT_CONFIG_FILE.

---

Run demo init script

/opt/kolla-venv/share/kolla-ansible/init-runonce

Purpose:
Creates demo resources such as a sample image, network, subnet, router and security group.

Use this only for test/evaluation environments. The docs warn that init-runonce is optional and may conflict with customised deployments.

---

4. Standard deployment workflow

For a clean multinode deployment:

source /opt/kolla-venv/bin/activate
export KOLLA_INVENTORY=/etc/kolla/multinode

kolla-ansible install-deps
kolla-genpwd

vi /etc/kolla/globals.yml
vi "$KOLLA_INVENTORY"

ansible -i "$KOLLA_INVENTORY" all -m ping
ansible-inventory -i "$KOLLA_INVENTORY" --graph

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY"
kolla-ansible prechecks -i "$KOLLA_INVENTORY"
kolla-ansible pull -i "$KOLLA_INVENTORY"
kolla-ansible deploy -i "$KOLLA_INVENTORY"
kolla-ansible post-deploy -i "$KOLLA_INVENTORY"

Then:

export OS_CLIENT_CONFIG_FILE=/etc/kolla/clouds.yaml
openstack --os-cloud admin service list

or, if you have an admin RC file:

source /etc/kolla/admin-openrc.sh
openstack service list

---

5. Configuration commands

Edit main configuration

vi /etc/kolla/globals.yml

Purpose:
Main Kolla-Ansible deployment configuration file.

The 2025.2 docs describe globals.yml as the main Kolla-Ansible configuration file, normally stored at /etc/kolla/globals.yml.

Typical core settings:

kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "2025.2"

network_interface: "ens18"
neutron_external_interface: "ens19"
kolla_internal_vip_address: "192.168.1.50"

The quickstart states that network_interface, neutron_external_interface, and kolla_internal_vip_address are key networking settings.

---

Use multiple globals files

mkdir -p /etc/kolla/globals.d
vi /etc/kolla/globals.d/cinder.yml
vi /etc/kolla/globals.d/neutron.yml
vi /etc/kolla/globals.d/gpu.yml

Example:

# /etc/kolla/globals.d/cinder.yml
enable_cinder: "yes"
enable_cinder_backup: "yes"

Purpose:
Keeps service-specific configuration isolated rather than overloading one huge globals.yml.

The 2025.2 docs state that Kolla-Ansible can automatically include multiple *.yml files placed under /etc/kolla/globals.d/.

---

Generate configuration only

kolla-ansible genconfig -i "$KOLLA_INVENTORY"

Purpose:
Renders Kolla service configuration files without fully deploying or restarting services.

Useful for checking generated config before a deployment or reconfigure.

---

Reconfigure all services

kolla-ansible reconfigure -i "$KOLLA_INVENTORY"

Purpose:
Applies configuration changes across the deployment and restarts containers where required.

Use this after changing /etc/kolla/globals.yml, /etc/kolla/globals.d/*.yml, or custom service configuration.

---

Reconfigure one service group

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags nova

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags neutron

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags horizon

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,horizon

Purpose:
Limits reconfiguration to a specific Kolla role or service group.

For example, your Horizon fix pattern should be:

source /opt/kolla-venv/bin/activate
kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,horizon

---

6. Certificate and TLS commands

Generate test certificates

kolla-ansible certificates -i "$KOLLA_INVENTORY"

Purpose:
Generates Kolla-Ansible test CA/certificates for TLS-enabled deployments.

The 2025.2 TLS guide states that the certificates command generates a private test Certificate Authority and signs certificates for enabled VIPs.

Typical TLS settings:

kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_enable_tls_backend: "yes"
kolla_copy_ca_into_containers: "yes"

The same TLS guide describes internal, external and backend TLS as distinct layers around the VIP/HAProxy/API path.

---

Reconfigure HAProxy and APIs after certificate changes

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags certificates,haproxy

or:

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,keystone,nova,neutron,glance,placement

Purpose:
Applies certificate changes to front-end routing and API services.

---

7. Service-specific deployment and reconfiguration

Enable Cinder later

Create a separate globals file:

mkdir -p /etc/kolla/globals.d
vi /etc/kolla/globals.d/cinder.yml

Example:

enable_cinder: "yes"
enable_cinder_backup: "yes"

Then run:

kolla-ansible prechecks -i "$KOLLA_INVENTORY" --tags cinder
kolla-ansible pull -i "$KOLLA_INVENTORY" --tags cinder
kolla-ansible deploy -i "$KOLLA_INVENTORY" --tags cinder

or:

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags cinder

Purpose:
Adds or reconfigures Block Storage without touching the whole cloud unnecessarily.

---

Reconfigure Nova

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags nova

Use after changes to:

nova.conf overrides
compute service settings
PCI passthrough settings
libvirt/Nova config
Placement/Nova integration

---

Reconfigure Neutron

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags neutron

Use after changes to:

ML2 config
Open vSwitch settings
external interface config
provider networks
DNS/domain settings
metadata agent settings

---

Reconfigure Horizon

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags horizon

For Horizon behind HAProxy:

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,horizon

---

Reconfigure Keystone

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags keystone

Use after changes to:

Keystone policy
Fernet/token configuration
domain configuration
TLS settings
authentication backends

---

Reconfigure Glance

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags glance

Use after changing image backend configuration, such as filesystem, Ceph/RBD, Swift, or external storage.

---

Reconfigure Placement

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags placement

Use after Placement API or policy changes.

---

Reconfigure HAProxy only

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy

Use after VIP, endpoint, TLS or service listener changes.

---

8. Operations commands

Check OpenStack service containers

docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' | sort

Purpose:
Kolla-Ansible deploys containers, so Docker/Podman state is the first operational check.

The troubleshooting guide says container state can be checked with docker ps -a; it also notes that Kolla logs are available under /var/log/kolla/SERVICE_NAME and stdout logs can be checked with docker logs <container-name>.

---

Stop all OpenStack services

kolla-ansible stop -i "$KOLLA_INVENTORY"

Purpose:
Stops OpenStack containers managed by Kolla-Ansible.

Use carefully. This is disruptive.

---

Stop a limited set of hosts

kolla-ansible stop -i "$KOLLA_INVENTORY" --limit compute

kolla-ansible stop -i "$KOLLA_INVENTORY" --limit gpu

Purpose:
Limits the operation to a host or Ansible group.

---

Deploy or reconfigure only limited hosts

kolla-ansible deploy -i "$KOLLA_INVENTORY" --limit gpu

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --limit cmp

kolla-ansible pull -i "$KOLLA_INVENTORY" --limit control

Purpose:
Targets specific nodes when adding, repairing, or updating a subset of the cloud.

The adding/removing-hosts guide shows bootstrap-servers, pull, and deploy with --limit when adding new controllers or compute nodes.

---

Batch bootstrap to reduce control-plane disruption

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" --limit controller0,compute[0-1]
kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" --limit controller1,compute[2-3]
kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" --limit controller2,compute[4-5]

or:

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" -e kolla_serial=30%

Purpose:
Avoids restarting Docker on too many control-plane nodes at once during subsequent bootstraps.

The bootstrap documentation explicitly recommends batching with --limit or kolla_serial when rerunning bootstrap on an already deployed cloud, because Docker package updates can restart Docker and containers.

---

9. Upgrade and image refresh commands

Pull images for current configured release

kolla-ansible pull -i "$KOLLA_INVENTORY"

Purpose:
Refreshes image cache before deployment, reconfiguration or upgrade.

---

Upgrade OpenStack

kolla-ansible upgrade -i "$KOLLA_INVENTORY"

Purpose:
Runs the Kolla-Ansible upgrade workflow after you have moved configuration and images to the target release.

A practical high-level upgrade pattern is:

source /opt/kolla-venv/bin/activate
pip install -U git+https://opendev.org/openstack/kolla-ansible@stable/2025.2

kolla-ansible install-deps
kolla-ansible prechecks -i "$KOLLA_INVENTORY"
kolla-ansible pull -i "$KOLLA_INVENTORY"
kolla-ansible upgrade -i "$KOLLA_INVENTORY"
kolla-ansible post-deploy -i "$KOLLA_INVENTORY"

---

Upgrade limited hosts or services

kolla-ansible upgrade -i "$KOLLA_INVENTORY" --limit compute

kolla-ansible upgrade -i "$KOLLA_INVENTORY" --tags nova

Purpose:
Useful for careful staged upgrades, but validate service-specific upgrade support before relying on this in production.

---

10. Add or remove node workflows

Add a new compute node

Update the inventory first:

vi "$KOLLA_INVENTORY"

Then:

ansible -i "$KOLLA_INVENTORY" new-compute-host -m ping

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" --limit new-compute-host
kolla-ansible prechecks -i "$KOLLA_INVENTORY" --limit new-compute-host
kolla-ansible pull -i "$KOLLA_INVENTORY" --limit new-compute-host
kolla-ansible deploy -i "$KOLLA_INVENTORY" --limit new-compute-host

Purpose:
Prepares and deploys OpenStack services onto a new compute node.

The adding-hosts guide gives this pattern: use bootstrap-servers, then pull, then deploy, with --limit for new hosts.

---

Add a new controller

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" --limit control
kolla-ansible pull -i "$KOLLA_INVENTORY" --limit new-controller
kolla-ansible deploy -i "$KOLLA_INVENTORY" --limit control

Purpose:
Adds a controller while ensuring controller-wide clustered services such as RabbitMQ and HA components have consistent host mappings.

The docs warn that when adding controllers, if using --limit, all controllers should be included for deployment in relevant steps.

---

11. Backup and restore commands

Enable MariaDB backup

In /etc/kolla/globals.yml or /etc/kolla/globals.d/mariadb.yml:

enable_mariabackup: "yes"

Then:

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags mariadb

The MariaDB backup guide says backup support requires enable_mariabackup: "yes" and a MariaDB reconfiguration.

---

Take a full MariaDB backup

kolla-ansible mariadb_backup -i "$KOLLA_INVENTORY"

Purpose:
Runs a full hot backup of the MariaDB/Galera database.

The 2025.2 MariaDB backup guide states that Kolla-Ansible uses Mariabackup for hot backups without database/cloud downtime.

---

Take an incremental MariaDB backup

kolla-ansible mariadb_backup -i "$KOLLA_INVENTORY" --incremental

Purpose:
Runs an incremental backup after a full backup.

---

MariaDB recovery

kolla-ansible mariadb-recovery -i "$KOLLA_INVENTORY" \
  -e mariadb_recover_inventory_name=controller1

Purpose:
Runs MariaDB recovery for a multinode deployment, pointing at the first node of the cluster.

The MariaDB restore documentation gives this command form for multinode recovery.

---

12. Container engine migration

Migrate Docker ↔ Podman

First edit:

kolla_container_engine: podman

Then run:

kolla-ansible migrate-container-engine -i "$KOLLA_INVENTORY"

Purpose:
Migrates a deployed OpenStack cloud from Docker to Podman, or from Podman to Docker.

The 2025.2 advanced configuration guide says Kolla-Ansible supports both Docker and Podman and can migrate in both directions, but rolling migration is not supported and running VMs must be stopped first.

---

13. Destructive and recovery commands

Destroy deployment

kolla-ansible destroy -i "$KOLLA_INVENTORY"

In many environments, Kolla requires explicit confirmation for destructive actions. Treat this as a cloud-removal operation.

Purpose:
Removes deployed containers and related deployment state.

The troubleshooting guide states that, after certain failed deployments, the fastest recovery may be to remove the failed deployment with kolla-ansible destroy -i <inventory-file>.

---

Destroy and then redeploy

kolla-ansible destroy -i "$KOLLA_INVENTORY"

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY"
kolla-ansible prechecks -i "$KOLLA_INVENTORY"
kolla-ansible pull -i "$KOLLA_INVENTORY"
kolla-ansible deploy -i "$KOLLA_INVENTORY"
kolla-ansible post-deploy -i "$KOLLA_INVENTORY"

Purpose:
Lab rebuild workflow when the environment is badly broken.

---

14. Troubleshooting commands

Run prechecks again

kolla-ansible prechecks -i "$KOLLA_INVENTORY"

Purpose:
First Kolla-native troubleshooting command after changing inventory or globals.yml.

---

Increase Ansible verbosity

kolla-ansible prechecks -i "$KOLLA_INVENTORY" -vv

kolla-ansible deploy -i "$KOLLA_INVENTORY" -vvv

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags neutron -vvv

Purpose:
Shows task-level detail when a playbook fails.

---

Check generated logs

readlink -f /var/log/kolla
ls -la /var/log/kolla

The troubleshooting guide states that the kolla_logs volume is linked to /var/log/kolla on the host and that service logs can be read under /var/log/kolla/SERVICE_NAME.

Examples:

ls -la /var/log/kolla/nova/
ls -la /var/log/kolla/neutron/
ls -la /var/log/kolla/keystone/
ls -la /var/log/kolla/horizon/

---

Enter fluentd container

docker exec -it fluentd bash

Purpose:
Troubleshoots logging collection.

The Kolla troubleshooting guide gives this as the way to examine logs via the logging container.

---

Check stdout logs

docker logs <container-name>

Example:

docker logs nova_compute
docker logs neutron_server
docker logs horizon
docker logs haproxy

The docs warn that many containers do not log much to stdout, so /var/log/kolla is usually more useful.

---

Check failed containers

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}' | sort

Focus on failed containers:

docker ps -a --format 'table {{.Names}}\t{{.Status}}' | grep -Ei 'Exited|Restarting|Dead|unhealthy'

---

Run OpenStack health checks from toolbox

docker exec kolla_toolbox bash -lc '
source /etc/kolla/admin-openrc.sh
openstack service list
openstack endpoint list
openstack compute service list
openstack network agent list
openstack hypervisor list
'

Purpose:
Separates “Kolla container deployment problem” from “OpenStack API/service registration problem”.

---

15. Advanced Ansible/Kolla targeting patterns

Limit to one host

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --limit gpu

kolla-ansible deploy -i "$KOLLA_INVENTORY" --limit cmp

---

Limit to a group

kolla-ansible pull -i "$KOLLA_INVENTORY" --limit compute

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --limit control

---

Run specific tags on specific hosts

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags nova --limit gpu

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags neutron --limit network

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,horizon --limit control

---

Skip tags

kolla-ansible deploy -i "$KOLLA_INVENTORY" --skip-tags common

Purpose:
Avoids selected task groups. Use carefully, because skipping foundational roles can produce inconsistent state.

---

Extra variables

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY" \
  -e ansible_user=ubuntu \
  -e ansible_python_interpreter=/usr/bin/python3

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" \
  -e kolla_serial=30%

Purpose:
Overrides variables at runtime.

---

16. Practical troubleshooting workflows

Workflow A: Deployment failed

source /opt/kolla-venv/bin/activate
export KOLLA_INVENTORY=/etc/kolla/multinode

ansible -i "$KOLLA_INVENTORY" all -m ping
ansible-inventory -i "$KOLLA_INVENTORY" --graph

kolla-ansible prechecks -i "$KOLLA_INVENTORY" -vv
docker ps -a
ls -la /var/log/kolla

Then inspect the failed service logs:

grep -RniE 'error|failed|traceback|exception' /var/log/kolla/nova/ | tail -50
grep -RniE 'error|failed|traceback|exception' /var/log/kolla/neutron/ | tail -50
grep -RniE 'error|failed|traceback|exception' /var/log/kolla/keystone/ | tail -50

---

Workflow B: Horizon change not taking effect

source /opt/kolla-venv/bin/activate

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,horizon

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep -Ei 'horizon|haproxy'
docker logs --tail 100 horizon
docker logs --tail 100 haproxy
docker exec horizon ss -tlnp
docker exec haproxy ss -tlnp

---

Workflow C: Neutron/networking issue

kolla-ansible prechecks -i "$KOLLA_INVENTORY" --tags neutron -vv

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags neutron

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep neutron
docker logs --tail 200 neutron_server
docker logs --tail 200 neutron_openvswitch_agent
docker logs --tail 200 neutron_dhcp_agent
docker logs --tail 200 neutron_l3_agent

Then:

source /etc/kolla/admin-openrc.sh
openstack network agent list
openstack router list
openstack port list

---

Workflow D: Nova / compute / GPU issue

kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags nova --limit gpu

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep nova
docker logs --tail 200 nova_compute
docker logs --tail 200 nova_scheduler
docker exec nova_libvirt virsh list --all

Then from OpenStack:

source /etc/kolla/admin-openrc.sh
openstack compute service list
openstack hypervisor list
openstack server list --all-projects --long

---

Workflow E: Image/tag mismatch after change

kolla-ansible pull -i "$KOLLA_INVENTORY"
kolla-ansible prechecks -i "$KOLLA_INVENTORY"
kolla-ansible deploy -i "$KOLLA_INVENTORY"

Use this when you have changed:

openstack_release
kolla_base_distro
container registry
image namespace
image tag suffix

---

17. Command summary by phase

Phase	Command	Purpose
Install deps	kolla-ansible install-deps	Install Ansible dependencies
Passwords	kolla-genpwd	Generate /etc/kolla/passwords.yml
Host prep	kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY"	Prepare hosts and Docker/container engine
Validation	kolla-ansible prechecks -i "$KOLLA_INVENTORY"	Check deploy readiness
Image staging	kolla-ansible pull -i "$KOLLA_INVENTORY"	Pull container images
Deployment	kolla-ansible deploy -i "$KOLLA_INVENTORY"	Deploy OpenStack containers
Client config	kolla-ansible post-deploy -i "$KOLLA_INVENTORY"	Generate clouds.yaml
Config render	kolla-ansible genconfig -i "$KOLLA_INVENTORY"	Generate config files
Apply config	kolla-ansible reconfigure -i "$KOLLA_INVENTORY"	Apply config changes
Upgrade	kolla-ansible upgrade -i "$KOLLA_INVENTORY"	Upgrade OpenStack deployment
Stop	kolla-ansible stop -i "$KOLLA_INVENTORY"	Stop OpenStack containers
Remove	kolla-ansible destroy -i "$KOLLA_INVENTORY"	Destroy deployment
Backup	kolla-ansible mariadb_backup -i "$KOLLA_INVENTORY"	Full MariaDB backup
Recovery	kolla-ansible mariadb-recovery -i "$KOLLA_INVENTORY"	Recover MariaDB
TLS	kolla-ansible certificates -i "$KOLLA_INVENTORY"	Generate test certificates

---

18. Best-practice command order for your homelab

For a fresh build:

source /opt/kolla-venv/bin/activate
export KOLLA_INVENTORY=/etc/kolla/multinode

kolla-ansible install-deps
kolla-genpwd

ansible -i "$KOLLA_INVENTORY" all -m ping
ansible-inventory -i "$KOLLA_INVENTORY" --graph

kolla-ansible bootstrap-servers -i "$KOLLA_INVENTORY"
kolla-ansible prechecks -i "$KOLLA_INVENTORY"
kolla-ansible pull -i "$KOLLA_INVENTORY"
kolla-ansible deploy -i "$KOLLA_INVENTORY"
kolla-ansible post-deploy -i "$KOLLA_INVENTORY"

For a normal config change:

vi /etc/kolla/globals.yml

kolla-ansible prechecks -i "$KOLLA_INVENTORY"
kolla-ansible reconfigure -i "$KOLLA_INVENTORY"

For a service-specific config change:

vi /etc/kolla/globals.d/neutron.yml

kolla-ansible prechecks -i "$KOLLA_INVENTORY" --tags neutron
kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags neutron

For Horizon specifically:

source /opt/kolla-venv/bin/activate
kolla-ansible reconfigure -i "$KOLLA_INVENTORY" --tags haproxy,horizon

For an image refresh:

kolla-ansible pull -i "$KOLLA_INVENTORY"
kolla-ansible deploy -i "$KOLLA_INVENTORY"

For troubleshooting:

kolla-ansible prechecks -i "$KOLLA_INVENTORY" -vv
docker ps -a
readlink -f /var/log/kolla
grep -RniE 'error|failed|traceback|exception' /var/log/kolla | tail -100

This gives you the practical Kolla-Ansible operator set for deployment, configuration, operations, service targeting, recovery, backup and troubleshooting on OpenStack 2025.2+.