Using TripleO composable roles to deploy compute nodes with heterogeneous hardware

There are a lot of situations where you may wish to deploy Openstack using TripleO onto heterogeneous hardware. Often you end up starting out with one type of hardware, but may purchase newer hardware later on, or you change hardware vendors, or you have different hardware that you wish to use to serve different workloads. This post will go over how to use the TripleO composable roles functionality introduced as part of the Newton release, to deploy Openstack with compute roles that match different hardware. This could in theory be applied to other roles (such as ceph and controller), but best practices for those roles advise having the same hardware in use.
In this example we want to use Lenovo Servers and Dell Servers as compute nodes, and each of these pieces of hardware have different configuration (networking, disk, etc). We are going to create two new compute roles, one called LenovoCompute, and another called DellCompute. We start by copying the file /usr/share/openstack-tripleo-heat-templates/roles_data.yaml to our local directory and adding in the extra roles like the following

# Add the following to the end of the file
- name: LenovoCompute
  HostnameFormatDefault: '%stackname%-lenovocompute-%index%'
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::NovaCompute
    - OS::TripleO::Services::NovaLibvirt
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::ComputeNeutronCorePlugin
    - OS::TripleO::Services::ComputeNeutronOvsAgent
    - OS::TripleO::Services::ComputeCeilometerAgent
    - OS::TripleO::Services::ComputeNeutronL3Agent
    - OS::TripleO::Services::ComputeNeutronMetadataAgent
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::NeutronSriovAgent
    - OS::TripleO::Services::OpenDaylightOvs
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts
- name: DellCompute
  HostnameFormatDefault: '%stackname%-dellcompute-%index%'
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::NovaCompute
    - OS::TripleO::Services::NovaLibvirt
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::ComputeNeutronCorePlugin
    - OS::TripleO::Services::ComputeNeutronOvsAgent
    - OS::TripleO::Services::ComputeCeilometerAgent
    - OS::TripleO::Services::ComputeNeutronL3Agent
    - OS::TripleO::Services::ComputeNeutronMetadataAgent
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::NeutronSriovAgent
    - OS::TripleO::Services::OpenDaylightOvs
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::VipHosts

Basically we take a copy of the Compute role from that file, and paste it twice, changing the name and HostnameFormatDefault for each role. Next we can create an environment file called compute_roles.yaml with the following content

resource_registry:
  OS::TripleO::DellCompute::Net::SoftwareConfig: nic-configs/dell-compute.yaml
  OS::TripleO::LenovoCompute::Net::SoftwareConfig: nic-configs/lenovo-compute.yaml
parameter_defaults:
  ComputeCount: 0
  DellComputeCount: 1
  LenovoComputeCount: 1
  DellComputeSchedulerHints:
    'capabilities:profile': 'dell-compute'
  LenovoComputeSchedulerHints:
    'capabilities:profile': 'lenovo-compute'
  DellComputeExtraConfig:
    nova::cpu_allocation_ratio: 24
  LenovoComputeExtraConfig:
    nova::cpu_allocation_ratio: 16

If we look at this file we can see it does a few things. First it defines separate nic config files for each of the different node types. This only takes effect if you are using network isolation, and allows you to have complete different network topologies for the different node types. Depending on how you structure the os-net-config settings in those files, you can use different nic, bridge, bond, and naming layouts for the different node types. More information on this is available here. Next it defines the node count for the original Compute role, as well as our new roles. You can change this to the number you desire. Below this is the profile matching we use to tell heat and ironic how to apply which role to which baremetal nodes. More information on that is here. Finally we use the ExtraConfig interface for each role to supply some node specific hiera data to those nodes. In this case, because the nodes have different cpu/memory/disk topologies, we want to change the nova cpu_allocation_ratio for an appropriate value for each hardware type.
We now need to update the nodes in ironic to define which node is to have which profile (and thus which tripleo role) applied to it. We can do this with

openstack baremetal node set --property capabilities=profile:lenovo-compute,boot_option:local $uuid_of_lenovo_compute
openstack baremetal node set --property capabilities=profile:dell-compute,boot_option:local $uuid_of_dell_compute

Now all we need to do is run our deploy command passing in the appropriate environment and roles file

openstack overcloud deploy -e compute_roles.yaml -r roles_data.yaml

And that’s it! This simple example shows how with different roles you can have different network configuration, and different Openstack settings depending on the hardware you are using. That is not the only way you can decide to use roles to define different node types. You might want different roles based of hardware location, or different owner, or failover domain, or any other scenario you can come up with.

Article written by

2 Responses

  1. Bah
    Bah at |

    How would you do the extra config for a more complex type like json type such as pci passthrough whitelist? I have hosts with different interface namings (ens1f0, enp129s0f0 etc) and want to enable sriov on all

Comments are closed.