Quick OpenStack Charm Test with Zaza

I was investigating a bug recently that appeared to be a race condition causing RabbitMQ server to occasionally fail to cluster when the charm was upgraded. To attempt to reproduce the bug I decided to use Python library zaza.

Firstly I created a tests & bundles directory:

$ mkdir -p tests/bundles/

Then added a bundle in tests/bundles/ha.yaml :

series: xenial
applications:
  rabbitmq-server:
    charm: cs:rabbitmq-server
    constraints: mem=1G
    num_units: 3

Once the deployment is complete I needed to test upgrading the charm so the next step was to define a test in tests/tests_rabbit_upgrade.py to do this:

#!/usr/bin/env python3

import unittest
import zaza.model

class UpgradeTest(unittest.TestCase):

    def test_upgrade(self):
        zaza.model.upgrade_charm(
            'rabbitmq-server',
            switch='cs:~openstack-charmers-next/rabbitmq-server-343')
        zaza.model.block_until_all_units_idle()

The last step is to tell Zaza what to do in a test run and this is done in the tests/tests.yaml.

tests:
  - tests.tests_rabbit_upgrade.UpgradeTest
configure:
  - zaza.charm_tests.noop.setup.basic_setup
smoke_bundles:
  - ha

This tells Zaza which bundle to run for smoke tests and which test(s) to run. There is no configuration step needed but at the moment zaza expects one so it is pointed at a no-op method. Obviously you need to have zaza installed which can be done in a virtualenv:

$ virtualenv -q -ppython3 venv3                                                                                                   
Already using interpreter /usr/bin/python3                                                                                                                      
$ source venv3/bin/activate                                                                                                      
(venv3) $ pip install git+https://github.com/openstack-charmers/zaza.git#egg=zaza

Zaza will generate a new model for each run and does not bring the new model into focus so it is safe to kick of the test run as many times as you like in parallel.

(venv3) $ functest-run-suite --smoke --keep-model &> /tmp/run.$(uuid -v1) &
[1] 4768
(venv3) $ functest-run-suite --smoke --keep-model &> /tmp/run.$(uuid -v1) &
[2] 4790
(venv3) $ functest-run-suite --smoke --keep-model &> /tmp/run.$(uuid -v1) &
[3] 4795
(venv3) $ functest-run-suite --smoke --keep-model &> /tmp/run.$(uuid -v1) &
[4] 4811

All four deployments and tests ran in parallel. Below is an example output:

$ cat /tmp/run.ed5fe80c-6045-11e9-83b5-77a3728f6280 
Controller: gnuoy-serverstack

Model              Cloud/Region             Type       Status      Machines  Cores  Access  Last connection
controller         serverstack/serverstack  openstack  available          1      4  admin   just now
default            serverstack/serverstack  openstack  available          0      -  admin   2018-11-13
zaza-310115a7b9f9  serverstack/serverstack  openstack  available          0      -  admin   just now
zaza-98856c9f8ec5  serverstack/serverstack  openstack  available          0      -  admin   just now
zaza-9c013d3f9ec9  serverstack/serverstack  openstack  available          0      -  admin   never connected

INFO:root:Deploying bundle './tests/bundles/ha.yaml' on to 'zaza-310115a7b9f9' model
Resolving charm: cs:rabbitmq-server
Executing changes:
- upload charm cs:rabbitmq-server-85 for series xenial
- deploy application rabbitmq-server on xenial using cs:rabbitmq-server-85
- add unit rabbitmq-server/0 to new machine 0
- add unit rabbitmq-server/1 to new machine 1
- add unit rabbitmq-server/2 to new machine 2
Deploy of bundle completed.
INFO:root:Waiting for environment to settle
INFO:root:Waiting for a unit to appear
INFO:root:Waiting for all units to be idle
INFO:root:Checking workload status of rabbitmq-server/0
INFO:root:Checking workload status message of rabbitmq-server/0
INFO:root:Checking workload status of rabbitmq-server/1
INFO:root:Checking workload status message of rabbitmq-server/1
INFO:root:Checking workload status of rabbitmq-server/2
INFO:root:Checking workload status message of rabbitmq-server/2
INFO:root:OK
INFO:root:## Running Test tests.tests_rabbit_upgrade.UpgradeTest ##
test_upgrade (tests.tests_rabbit_upgrade.UpgradeTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 87.692s

OK

Finally, because the --keep-model switch was used all four models are still available for any post test inspection:

$ juju list-models
Controller: gnuoy-serverstack

Model              Cloud/Region             Type       Status     Machines  Cores  Access  Last connection
controller         serverstack/serverstack  openstack  available         1      4  admin   just now
default*           serverstack/serverstack  openstack  available         0      -  admin   2018-11-13
zaza-310115a7b9f9  serverstack/serverstack  openstack  available         3      6  admin   6 minutes ago
zaza-98856c9f8ec5  serverstack/serverstack  openstack  available         3      6  admin   6 minutes ago
zaza-9c013d3f9ec9  serverstack/serverstack  openstack  available         3      6  admin   5 minutes ago
zaza-a1b9f911850b  serverstack/serverstack  openstack  available         3      6  admin   6 minutes ago

To read more about zaza head over to https://zaza.readthedocs.io/en/latest/