This article documents our team's research journey exploring Ansible AWX as an infrastructure automation orchestration platform — from initial deployment and OpenStack integration to air-gap installation.

What is Ansible AWX?

Ansible AWX is open-source version of Red Hat Ansible Automation Platform, providing a Web UI to interactively manage Ansible resources. With AWX, teams can run playbooks, manage inventories, schedule jobs, and handle credentials, all through a browser without needing to SSH into a terminal.

Since version 18.0, AWX is recommended to be deployed using an Operator on top of a Kubernetes platform.

Stage 1: Deploying AWX on K3s

Why K3s?

None

K3s is a lightweight Kubernetes distribution built by Rancher that retains the full functionality of Kubernetes while consuming far fewer resources compared to vanilla Kubernetes. It's well-suited for lab environments with limited specs (minimum 4 cores / 8 GB RAM for a single-node AWX setup).

Deployment Steps

Install K3s:

curl -sfL https://get.k3s.io | sh -

Install AWX Operator:

git clone https://github.com/ansible/awx-operator.git
cd awx-operator
git checkout 2.19.1
make deploy

Create an AWX Instance:

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-instance
  namespace: awx
spec:
  service_type: NodePort

AWX is then accessible via NodePort, and the admin password can be retrieved with:

kubectl get secret -n awx awx-instance-admin-password -o jsonpath="{.data.password}" | base64 --decode

Stage 2: Dynamic Inventory from OpenStack

Problem: DNS Resolution Failure

When attempting to sync dynamic inventory from OpenStack, AWX can failed because pods inside Kubernetes couldn't resolve internal domains. The fix was to update the CoreDNS ConfigMap:

kubectl edit configmaps coredns -n kube-system

Add the required host entries to the NodeHosts block. After restarting CoreDNS, nslookup from inside the pod succeeded and inventory sync completed successfully.

Setting Up OpenStack Dynamic Inventory

Create a custom Credential Type for OpenStack with input fields: username, password, project_name, auth_url, and region_name. The Injector Configuration maps these fields to OS_* environment variables.

For multi-project support, add the following to the inventory Source Variables:

plugin: openstack.cloud.openstack
cloud: myopenstack
expand_hostvars: true
fail_on_errors: true
all_projects: true

Stage 3: Handling Dynamic SSH Users

One of the main challenges was that each OpenStack instance could have a different default SSH user (ubuntu, centos, cloud-user) and a different SSH key. Our team developed several approaches:

Approach 1: Bash Script to Update ansible_user

The update_host.sh script reads an ip_user_map.json file and sends PATCH requests to the AWX API to update each host's variables accordingly.

Approach 2: Auto-Detection via Ansible Facts

Create two Job Templates — one for setup (collecting and storing ansible facts into fact storage), and another for running the main tasks using already-known user from stored facts.

- name: Set ansible_user dynamically
  set_fact:
    ansible_user: >-
      {{
        ansible_env.SUDO_USER |
        default(ansible_user_id, true) |
        default('ubuntu')
      }}

Approach 3: Detection via OpenStack Image Name

Use image_name metadata from an OpenStack volume to automatically determine the correct SSH user via regex_replace.

Stage 4: AWX Execution Nodes (Multi-Project)

Network Isolation Between Projects

Each OpenStack project has its own isolated network. AWX control plane cannot always reach instances across all projects. The solution: deploy a Receptor-based Execution Node in each project as an execution agent.

How It Works

AWX control plane → sends instructions via Receptor → Execution Node inside the target project runs playbook → results are returned to AWX.

OS Requirements for Execution Nodes

Ubuntu 22.04 (Jammy) or RHEL 9 is required because:

  • Podman is officially available starting from Focal 20.10+
  • Python >= 3.9 is already the base version
  • The latest Ansible versions support FQCN syntax

Installing Receptor

  1. Add a new instance in AWX UI and download bundle installer
  2. Install Ansible on execution node
  3. Edit inventory.yml inside bundle — update ansible_host and ansible_user
  4. Run installer playbook:
  5. Add DNS mapping for the execution node hostname in CoreDNS
  6. Perform a health check from the AWX dashboard
  7. Create an Instance Group and assign the execution node to that group

Stage 5: Manual Playbooks & Volume Mounting

To run playbooks stored manually (not from SCM/Git), AWX needs to be configured so its pods mount to the same directory on host.

spec:
  web_extra_volume_mounts: |
    - name: playbook-volume
      mountPath: /var/lib/awx/projects
  task_extra_volume_mounts: |
    - name: playbook-volume
      mountPath: /var/lib/awx/projects
  extra_volumes: |
    - name: playbook-volume
      hostPath:
        path: /data/projects
        type: Directory

All playbooks can then simply be stored in /data/projects on the AWX host VM.

Stage 6: Custom Execution Environment (EE)

If a playbook requires a collection not available in the default EE image (e.g., openstack.cloud), a custom EE image needs to be built using ansible-builder.

pip3 install ansible-builder
ansible-builder build -t ee-openstack:latest

The image then pushed to a registry and registered in AWX as a new Execution Environment.

Stage 7: Air-Gap Install (Offline Method)

In production environments isolated from the internet, the installation is done fully offline using:

  1. Registry Mirror (Docker Registry v2) to cache images from docker.io and quay.io
  2. K3s Airgap Install using a pre-downloaded k3s-airgap-images-amd64.tar.zst file
  3. registries.yaml configuration so K3s pulls images from the local mirror

The end result is a fully functional AWX installation running without any internet connection, using the local registry mirror as the image source.

Capacity & Forks Calculation

AWX calculates execution node capacity based on:

  • CPU: num_cores × 4 forks
  • RAM: ram_size_mb / 100 forks

The lower of two values is used. For an instance with 4 vCPU / 16 GB RAM, AWX can process up to 1,100 events/second and execute up to 137 forks simultaneously — more than enough to manage thousands of hosts at once.

Lessons Learned

Key takeaways from this research:

  • DNS is everything. The most common issue when deploying AWX on K8s always comes back to CoreDNS configuration.
  • SSH users must be dynamic. In a multi-OS OpenStack environment, a single-credential-for-all approach simply won't work.
  • One Execution Node per project is the best solution for network isolation across OpenStack projects.
  • Air-gap install requires thorough preparation — all dependencies must be available in local registry before starting the installation.
  • AWX Fact Storage is incredibly useful for caching host information so playbooks don't need to gather facts on every run.

Closing

Ansible AWX proves to be a powerful platform for managing infrastructure automation at scale. With its operator-based approach on Kubernetes, AWX provides high flexibility — from single-node deployments to multi-execution-node clustering in fully isolated enterprise environments.

This research is still ongoing for further exploration of advanced AWX features. Some case in production environment, we used AWX for automatically upgrade service in hundred of nodes with interactive dashboard that decrease implementation time duration become more fast and efficient.

Need Help with Your Automation Platform?

Implementing and managing an automation platform can be challenging. Starting from create design architecture, method of procedure implementing and managing your automation platform such as AWX for ready to use in production level.

At Boer Technology, we expertise in designing, implementing and managing automation platform system solution including AWX. Our team has experience with managing production level with a hundred of nodes, which will allow your environment to be managed centrally and efficiently.

Get in Touch

Ready to managing your platform automatically? Let's discuss how we can help:

References

Author

Managed Services Team — PT. Boer Technology

Tags: #AWX #AutomationPlatform #Ansible #OpenSource #DevOps #Kubernetes