Distributed-CI and InfraRed

Introduction

Red Hat OpenStack QE team maintains a tool to deploy and test OpenStack. This tool can deploy different types of topologies and is very modular. You can extend it to cover some new use-case. This tool is called InfraRed and is a free software and is available on GitHub.

The purpose of Distributed-CI (or DCI) is to help OpenStack partners to test new Red Hat OpenStack (RHOSP) releases before they are published. This allows them to train on new releases, identify regression or prepare new driver ahead of time. In this article, we will explain how to integrate InfraRed with another too called Distributed-CI, or DCI.

InfraRed

InfraRed has been designed to be flexible and it can address numerous different use-cases. In this article, we will use it to prepare a virtual environment and driver a regular Red Hat OpenStack Platform 13 (OSP13) deployment on it.

InfraRed is covered by a complete documentation that we won’t copy past here. To summarize, once it’s installed, InfraRed exposes a CLI. This CLI gives the user the ability to create a workspace that will trace the state of the environment. The user can then trigger all the required steps to ultimately get a running OpenStack. In addition, InfraRed offers additional features through a plug-in system.

Distributed-CI

Global diagram of DCI

The partners use DCI to validate OpenStack on their labs. It’s a way to validate that they will still be able to use their gear with the next release. A DCI agent runs the deployment and is in charge of the communication with Red Hat. They then have to provide a set of scripts to deploy OpenStack on it automatically. These scripts will be used during the deployment.

DCI can be summarized with the following list of actions:

  1. Red Hat exposes the last internal snapshots of the product on the DCI
  2. Partner’s DCI agent pulls the last snapshot and deploys it internally using the local configuration and deployment scripts
  3. Partner’s DCI agent runs the tests and sends back the final result to DCI.

Deployment of the lab

For this article, we will use a libvirt hypervisor to virtualize our lab. The hypervisor can be based either on RHEL7 or a CentOS7.

The network configuration

In this tutorial, we will rely on libvirt ‘default’ network. This network uses the 192.168.122.0 range. 192.168.122.1 is our hypervisor. The IP addresses of the other VM will by dynamical and InfraRed will create some additional networks for you. We also use the hypervisor public IP which is `192.168.1.40.

Installation of the Distributed-CI agent for OpenStack

The installation of DCI agent is covered by its own documentation. All the steps are rather simple as soon as the partner has a host to run the agent that matches DCI requirements. This host is called the jumpbox in DCI jargon. In this document, the jumpbox is also the hypervisor host.

In the rest of this document will assume you have an admin access to a DCI project, that you created the remoteci on http://www.distributed-ci.io and that you have deployed the agent on your jumpbox with the help if its installation guide. To validate everything, you should be able to list the remoteci of your tenant with the following command.

# source /etc/dci-ansible-agent/dcirc.sh
# dcictl remoteci-list
+--------------------------------------+--------------+--------+------------------------------------------------------------------+--------+--------------------------------------+--------------------------------------+
|                  id                  |     name     | state  |                            api_secret                            | public |               role_id                |               team_id                |
+--------------------------------------+--------------+--------+------------------------------------------------------------------+--------+--------------------------------------+--------------------------------------+
| e86ab5ba-695c-4437-b163-261e20b20f56 | FutureTown | active | something |  None  | e5e20d68-bbbe-411c-8be4-e9dbe83cc74e | 2517154c-46b4-4db9-a447-1c89623cc00a |
+--------------------------------------+--------------+--------+------------------------------------------------------------------+--------+--------------------------------------+--------------------------------------+

So far so good, we can now start the agent for the very first time with:

# systemctl start dci-ansible-agent --no-block
# journalctl -exf -u dci-ansible-agent

The agent pulls the bits from Red Hat and uses the jumpbox to expose them. Technically speaking, it’s a Yum repository in /var/www/html and a image registry on port 5000. These resources need to be consumed during the deployment. Since we don’t have any configuration yet, the run will fail. It’s time to fix that and prepare our integration with InfraRed.

One of the crucial requirement is the set of scripts that will be used to deploy OpenStack. Those scripts are maintained by the user. They will be called by the agent through a couple of Ansible playbooks:

  • hooks/pre-run.yml: This playbook is the very first one to called on the jumpbox. It’s the place where the partner can, for instance, fetch the last copy of the configuration.
  • hooks/running.yml: This is the place where the automation will be called. Most of the time, it’s a couple of extra Ansible tasks that will call a script or include another playbook.

Preliminar configuration

Security, firewall and SSH keypair

Some services like Apache will be exposed without any restriction. This is why we assume the hypervisor is on a trusted network.

We take the freedom to disable firewalld to simplify the whole process. Please do:

# systemctl stop firewalld
# systemctl disable firewalld

InfraRed interacts with the hypervisor using SSH. Just a reminder, in our case, the hypervisor is the local machine. To keep the whole setup simple, we share the same SSH key for the root and dci-ansible-agent users:

# ssh-keygen
# mkdir -p /var/lib/dci-ansible-agent/.ssh
# cp /root/.ssh /var/lib/dci-ansible-agent/.ssh
# chown -R dci-ansible-agent:dci-ansible-agent /var/lib/dci-ansible-agent/.ssh
# chmod 700 /var/lib/dci-ansible-agent/.ssh
# chmod 600 /var/lib/dci-ansible-agent/.ssh/*
# restorecon /var/lib/dci-ansible-agent/.ssh

You can validate everything work fine with:

# su - dci-ansible-agent
$ ssh root@localhost id

Libvirt

We will deploy OpenStack on our libvirt hypervisor with the Virsh provisioner.

# yum install libvirt
# systemctl start libvirtd
# systemctl enable libvirtd

Red Hat Subscription Manager configuration (RHSM)

InfraRed uses the RHSM during the deployment to register the nodes and pull the last RHEL updates. It loads the credentials from a little YAML file that you can store in the /etc/dci-ansible-agent directory with the other files:

# cat /etc/dci-ansible-agent/cdn_creds.yml
username: gleboude@redhat.com
password: 9328878db3ea4519912c36525147a21b
autosubscribe: yes

RHEL guest image

InfraRed needs a RHEL guest image to prepare the nodes. It tries hard to download it by itself, thanks InfraRed… But the default location is https://url.corp.redhat.com/rhel-guest-image-7-5-146-x86-64-qcow2 which is unlikely to match your environment. Got on https://access.redhat.com/downloads and download the last RHEL guest image. The file should be stored here on your hypervisor: /var/lib/libvirt/images/rhel-guest-image-7-5-146-x86-64-qcow2. The default image name will probably change in the future, you can list the default values for the driver with the infrared (or ir) command:

# su - dci-ansible-agent
$ source .venv/bin/activate
$ ir virsh --help

Configure the agent for InfraRed

All the configuration files of this example are available on GitHub.

Run bootstrap (pre-run.yml)

First, we want to install InfraRed dependencies and prepare a virtual environment. These steps will be done with the pre-run.yml.

---
- name: Install the RPM that InfraRed depends on
  yum:
    name: '{{ item }}'
    state: present
  with_items:
  - git
  - python-virtualenv
  become: True

We pull InfraRed directly from its Git repository using Ansible’s git module:

- name: Wipe any existing infrared virtualenv
  file:
    path: ~/infrared
    state: absent
- name: Pull the last InfraRed version
  git:
    repo: https://github.com/openstack-redhat/infrared.git
    dest: /var/lib/dci-ansible-agent/infrared
    version: master

Finally, we prepare a Python virtual environment to preserve the integrity of the system and we install InfraRed in it.

- name: Wipe any existing infrared virtualenv
  file:  ~/infrared/.venv
    state: absent
- name: Install InfraRed in a fresh virtualenv
  shell: |
    cd ~/infrared
    virtualenv .venv && source .venv/bin/activate
    pip install --upgrade pip
    pip install --upgrade setuptools
    pip install .

As mentioned above, the agent is called by the dci-ansible-agent user, we have to ensure everything is done in its home directory.

- name: Enable the InfraRed plugins that we will use during the deployment
  shell: |
    cd ~/infrared
    source .venv/bin/activate
    infrared plugin add plugins/virsh
    infrared plugin add plugins/tripleo-undercloud
    infrared plugin add plugins/tripleo-overcloud

Before we start anything, we do a cleanup of the environment. For that, we rely on InfraRed. Its virsh plugin can remove all the existing resources thinks to the --cleanup argument:

- name: Clean the hypervisor
  shell: |
    cd ~/infrared
    source .venv/bin/activate
    infrared virsh \
      --host-address 192.168.122.1 \
      --host-key $HOME/.ssh/id_rsa \
      --cleanup True

Be warned, InfraRed removes all the existing VM, network and storages from your hypervisor.

Hosts deployment (running.yml)

As mentioned before, the running.yml is actually the place where the deployment is actually done. We ask InfraRed to prepare our hosts:

- name: Prepare the hosts
  shell: |
    cd ~/infrared
    source .venv/bin/activate
    infrared virsh \
      --host-address 192.168.122.1 \
      --host-key $HOME/.ssh/id_rsa \
      --topology-nodes undercloud:1,controller:1,compute:1

Undercloud deployment (running.yml)

We can now deploy the Undercloud:

- name: Install the undercloud
  shell: |
    cd ~/infrared
    source .venv/bin/activate
    infrared tripleo-undercloud \
      --version 13 \
      --images-task rpm \
      --cdn /etc/dci-ansible-agent/cdn_creds.yml \
      --repos-skip-release True \
      --repos-url http://192.168.1.40/dci_repo/dci_repo.repo

At this stage, our libvirt virtual machines are ready and one of them host the undercloud. All these machines have a floating IP. InfraRed keeps the machines names up to date in /etc/hosts. We rely on that to get the undercloud IP address:

- name: Registry InfraRed's undercloud-0 IP
  set_fact: undercloud_ip="{{ lookup('pipe', 'getent hosts undercloud-0').split()[0]}}"

You can also use InfraRed to interact with all these hosts with a dynamic IP:

# su - dci-ansible-agent
$ cd ~/infrared
$ source .venv/bin/activate
$ ir ssh undercloud-0

Here ir is an alias for the infrared command. In both cases, it’s pretty cool, InfraRed did all the voodoo for us.

Overcloud deployment (running.yml)

It’s time to run the final step of our deployment.

- name: Deploy the overcloud
 shell: |
     cd ~/infrared
    source .venv/bin/activate
    infrared tripleo-overcloud \
      --deployment-files virt \
      --version 13 \
      --introspect yes \
      --tagging yes \
      --deploy yes \
      --post yes \
      --containers yes \
      --registry-skip-puddle yes \
      --registry-undercloud-skip yes \
      --registry-mirror 192.168.1.40:5000 \
      --registry-tag latest \
      --registry-namespace rhosp13 \
      --registry-prefix openstack- \
      --vbmc-host undercloud \
      --ntp-server 0.rhel.pool.ntp.org

Here we pass some extra arguments to accommodate InfraRed:

  • --registry-mirror: we don’t want to use the images from Red Hat. Instead, we will pick the ones delivered by DCI. Here 192.168.1.40 is the first IP address of our jumpbox. It’s the one the agent use when it deploys the image registry. Use the following command to validate you use the correct address: cat /etc/docker-distribution/registry/config.yml|grep addr
  • --registry-namespace and --registry-prefix: our images name start with /rhosp13/openstack-.
  • --vbmc-host undercloud: During the Overcloud installation, TripleO uses Ironic for the node provisioning. Ironic interacts with the nodes through a Virtual BMC server. By default InfraRed install it on the hypervisor, in our case we prefer to keep it clean. This is why we target the undercloud instead.

The virtual BMC instances will look like that on the undercloud:

[stack@undercloud-0 ~]$ ps aux|grep bmc
stack     4315  0.0  0.0 426544 15956 ?        Sl   13:19   0:00 /usr/bin/python2 /usr/bin/vbmc start controller-2
stack     4383  0.0  0.0 426544 15952 ?        Sl   13:19   0:00 /usr/bin/python2 /usr/bin/vbmc start controller-1
stack     4451  0.0  0.0 426544 15952 ?        Sl   13:19   0:00 /usr/bin/python2 /usr/bin/vbmc start controller-0
stack     4520  0.0  0.0 426544 15936 ?        Sl   13:19   0:00 /usr/bin/python2 /usr/bin/vbmc start compute-1
stack     4590  0.0  0.0 426544 15948 ?        Sl   13:19   0:00 /usr/bin/python2 /usr/bin/vbmc start compute-0
stack    10068  0.0  0.0 112708   980 pts/0    S+   13:33   0:00 grep --color=auto bmc

DCI lives

Let’s start the beast!

Ok, at this stage, we can start the agent. The standard way to trigger a DCI run is through systemd:

# systemctl start dci-ansible-agent --no-block

A full run takes more than 2 hours, the --no-block argument above tells systemctl to give back the control to the shell. Even if the unit’s start-up is not completed yet.

You can follow the progress of your deployment either on the web interface: https://www.distributed-ci.io/ or with journalctl:

# journalctl -exf -u dci-ansible-agent

The CLI

DCI also comes with CLI interface that you can use directly on the hypervisor.

# source /etc/dci-ansible-agent/dcirc.sh
# dcictl job-list

This command can also give you an output in the JSON format. It’s handy when you want to reuse the DCI results in some script:

# dcictl --format json job-list --limit 1 | jq .jobs[].status
running

To conclude

I hope you enjoyed the article and this will help you to prepare your own configuration. Please, don’t hesitate to contact me if you have any question.

I would like to thanks François Charlier and the InfraRed team. François started the DCI InfraRed integration several months ago. He did a great job to resolve all the issues one by one with the help of the InfraRed team.

Article written by