What is Zuul
Zuul is a software developed by the OpenStack community. It was developed as an efficient gated commit system, allowing projects to merge patches only after they pass a series of tests. It reduces the probability of breaking the master branch, for instance when unit tests or functional tests no longer pass on the tip of master.
It also performs well for running jobs when specific events occur like when a patch is proposed, merged, … or even when a project is tagged.
In this blog post we’ll try to explain how Zuul works and clarify some concepts through simple examples:
- The independent pipeline
- The dependent pipeline
- The Zuul cloner
- The Cross project testing
To ease the understanding of this blog post you should be familiar with Gerrit and Jenkins.
We wrote this article with the intend to dive more into Zuul use cases and share these with the interested reader. We also wanted to improve Zuul integration inside our CI/CD platform Software Factory.
How does Zuul keep your master branch sane ?
Keeping the master branch sane is difficult when:
- validating a patch for a project takes a long time
- the amount of patches proposals submitted to a project is quite high
Zuul helps the project’s core reviewers (those who can approve the merge of a patch in master) to decide whether a patch can land into the master branch or not by ensuring the patch is always tested over the latest version of master prior to merging.
Zuul is designed to be coupled with a code review software like Gerrit. It allows code reviewers to validate patches on a project and to decide whether a patch is accepted (good to be merged on the master branch) or not (the patch needs more work or should be abandoned).
When the patch submission rate is high, then keeping patches rebased on master’s HEAD is difficult, but validating a patch on the HEAD is essential to keep the master branch sane. For example, between the moment a core reviewer decides to check a patch (A) and the moment he accepts it by submitting it to master, another reviewer may have merged another patch (B). In that situation, the tip of the master might end up in an unexpected, undesirable or even broken state because the first reviewer has just tested master HEAD + (A) and not master HEAD + (B) + (A).
Instead, Zuul listens to the Gerrit event stream and detects if a patch (we’ll call it (A)) was approved for merging by a core reviewer. If this is the case, Zuul runs the project test suite on top of the project master HEAD with patch (A) applied on it. If another patch (B) was being tested for the same project (therefore pending a merge) prior to the submision of patch (A), Zuul will run the test suite on HEAD + (B) + (A). Depending on the results, Zuul will notify Gerrit to merge the patch on master or to report a test failure. This behavior is handled by the dependent pipeline of Zuul (gate).
Furthermore, let’s say you have a functional test suite that takes at least 2 hours to run. Without a tool like Zuul you could at most merge 12 patches a day. Zuul increases this rate by running the tests in parallel while still respecting the order of acceptance of patches.
High level Zuul architecture
Zuul scheduler
This component listens to the Gerrit event stream and triggers actions to be performed depending on the events, based on a configuration file. The most important configuration file for Zuul is “/etc/zuul/layout.yaml”; it defines various settings such as:
- the available pipelines
- the conditions for a patch to enter in a pipeline
- which jobs to run within a pipeline for a given project or a set of projects
- how to report a job result
- custom actions to perform
Zuul merger
The merger’s purpose is to set temporary Git repositories and branches up in order to ease the preparation of jobs environments. When a patch is scheduled to be tested, the scheduler will ask the merger to prepare a temporary branch (ZUUL_REF) where the patches have been merged on the tip of master of one or more projects, depending on what is required for the job. Then Zuul scheduler will pass details about that temporary branch to the jobs runner, allowing it to fetch the proper environment to start a job.
Zuul cloner
This is a handy python script that helps to properly setup a job workspace with the patch(es) to be tested for one or more projects. The scheduler sets some environment variables when triggering the job runner to run a job. Then the job runner can use those variables when preparing the workspace. The cloner knows how to interpret those variables, so it can be called at the beginning of a job script. The cloner clones the master branch of every project needed from the main repository and fetches patch(es) from Zuul merger’s temporary repository’s ZUUL_REF branch.
Want to try out Zuul ?
In order to help you experiment easily with Zuul we have setup a Dockerfile that builds a ready-to-use container with all the components you need to get started with commit gating, in case you don’t have your own setup or time to deploy one properly.
Checkout the exzuul project and follow the README:
$ git clone https://github.com/morucci/exzuul.git |
All examples below have been run on the exzuul container.
The container starts these main components:
- Gerrit (Code Review)
- Zuul (Gating)
- Zuul status page
- Jenkins (Job runner)
- Apache (with git-http-backend)
Below is the basic architecture running when you start the container. The diagram also gives an idea of components interactions.
In this blog post we define jobs in Jenkins using Jenkins Job Builder (JJB). You can also find an interesting blog post about it here.
Independent and dependent Pipelines
The dependent and independent pipelines follow different behaviors:
- The dependent pipeline is best used for commit gating (it merges patches)
- The independent pipeline can be used to run jobs that can usually be run independently, like these:
- Run tests to get early feedback on a patch, like smoke tests
- Run periodic jobs, like checking the availability of external dependencies
- Run post merge jobs, like building and uploading the project’s documentation
- Run jobs when a tag is created, like building and uploading a binary for the project
Actually you can have as many pipelines as you want and bind them on custom events that are going to occur on the Gerrit events stream. That means you can run jobs at any moment during the life cycle of a patch or a project.
The exzuul container bundles Zuul with the following pre-defined pipelines:
- check (Independent pipeline)
- gate (Dependent pipeline)
The independent pipeline
Below is the configuration of the independent pipeline named ‘check’ in /etc/zuul/layout.yaml.
- name: check
description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
failure-message: Build failed.
manager: IndependentPipelineManager
precedence: low
require:
open: True
current-patchset: True
trigger:
gerrit:
- event: patchset-created
- event: comment-added
comment: (?i)recheck
start:
gerrit:
verified: 0
success:
gerrit:
verified: 1
failure:
gerrit:
verified: -1
According to the trigger section, this pipeline run jobs if a new patch is submitted (gerrit event called "patchset-created") or if a comment containing the keyword "recheck" (gerrit event called "comment-added") is posted on a review thread. You can see that the pipeline reports a score of 0 on the "Verified" label when the job starts, -1 if the job fails and +1 if it succeed.
In order to test that pipeline we will add a small project called 'democlient' on Gerrit and we are going to run its unit tests through zuul and jenkins. As the pipeline is already defined, we just need to associate the project name with a pipeline and a job name. In order to do so, add the following in /etc/zuul/layout.yaml under the "projects" section:
- name: democlient
check:
- democlient-unit-tests
Validate layout.yaml and force Zuul to reload its configuration by running the following command on the container shell:
# zuul-server -l /etc/zuul/layout.yaml -t && kill -HUP $(pgrep zuul-server | head -1) |
Zuul is ready to ask the job runner (Jenkins) to run democlient-unit-tests inside the check pipeline, but we need to define that job on Jenkins first. To do that we use JJB (Jenkins Job Builder).
Edit /etc/jenkins_jobs/jobs/jjb.yaml and add the following:
- job:
name: democlient-unit-tests
defaults: global
builders:
- shell: |
env | grep ZUUL
zuul-cloner http://ci.localdomain:8080 $ZUUL_PROJECT
cd $ZUUL_PROJECT
./run_tests.sh
- project:
name: democlient
node: master
jobs:
- democlient-unit-tests
Submit the job on Jenkins via JJB by running the following command on the container shell:
# jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini update /etc/jenkins_jobs/jobs |
Now initialize the democlient project on the Gerrit UI. For the sake of simplicity we are going to use the default Admin account (already created on Gerrit) to perform user actions on Gerrit:
- Add your public ssh key to the Gerrit Admin account settings.
- Create a project called "democlient" using the Admin account on Gerrit. Be sure to check "create an empty commit" before creation.
Configure your local "democlient" repository for reviewing with gerrit, and push the initial code on "democlient" by running these commands on your host:
$ git clone https://github.com/morucci/democlient.git democlient_origin | |
$ git clone http://ci.localdomain:8080/democlient | |
$ cd democlient | |
$ git checkout -b"initial_code" | |
$ cp -R ../democlient_origin/* . | |
$ cat > .gitreview << EOF | |
[gerrit] | |
host=ci.localdomain | |
port=29418 | |
project=democlient | |
EOF | |
$ ssh-keygen -f "$HOME/.ssh/known_hosts" -R [ci.localdomain]:29418 | |
$ git config --add gitreview.username "admin" | |
$ git review -s | |
$ git add .gitreview * | |
$ git commit -m"push inital code" | |
$ git review |
You should see that the patch freshly submitted via git review has been "Verified" by Zuul by giving it a +1 vote.
Your patch has entered the check pipeline because it matches the trigger conditions defined inside the pipeline.
The job related to the project and the check pipeline begins with running zuul-cloner in order to prepare the workspace. We do not rely on any Jenkins plugin to prepare the environment; instead zuul-cloner receives a bunch of environment variables that it uses to fetch the patch.
The most important variables are ZUUL_PROJECT, ZUUL_URL, and ZUUL_REF. Zuul-cloner clones democlient from Gerrit at the current tip of master branch, then fetches ZUUL_REF from democlient zuul-merger's temporary repository. Indeed the merger has been instructed to prepare a specific branch called ZUUL_REF where the change has been applied.
Below are the logs of the job console where you can see the output of the cloner.
...
+ env
ZUUL_PROJECT=democlient
ZUUL_BRANCH=master
ZUUL_URL=http://ci.localdomain/p
ZUUL_CHANGE=1
ZUUL_CHANGES=democlient:master:refs/changes/01/1/2
ZUUL_REF=refs/zuul/master/Z86a40a16c3064a9ca9f48d590d89e2b7
ZUUL_CHANGE_IDS=1,2
ZUUL_PIPELINE=check
ZUUL_COMMIT=03285f35e11a225af0a6da55d647871ece06cfde
ZUUL_PATCHSET=2
ZUUL_UUID=1d710aa40eea46198143513241f06309
+ zuul-cloner http://ci.localdomain:8080 democlient
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/democlient-unit-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: democlient - /var/lib/jenkins/workspace/democlient-unit-tests/democlient
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo democlient from upstream http://ci.localdomain:8080/democlient
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared democlient repo with commit 03285f35e11a225af0a6da55d647871ece06cfde
INFO:zuul.Cloner:Prepared all repositories
+ cd democlient
+ ./run_tests.sh
...
We can verify inside zuul-merger's democlient repository that the tip of ZUUL_REF corresponds to your commit. Run the following command in the container shell:
# cd /var/lib/zuul/git/democlient/ | |
# git log --pretty=oneline --abbrev-commit $ZUUL_REF | |
03285f3 push inital code | |
406641a Initial empty repository |
Now let's say you or someone else submits a patch that depends on your previous, not yet merged patch. Zuul-merger will prepare a branch under a new ZUUL_REF that will include your previous patch and the new one. On the host shell:
$ cd democlient | |
$ git checkout master | |
$ git review -d <git id of the previous patch> | |
$ git checkout -b "improvement" | |
$ touch dummy && git add dummy && git commit -m"add dummy" && git review | |
# Respond yes to the question saying that it is ok you want to | |
# submit a commit that is dependent on one or more in-review. |
In the job console logs you can see that the variable ZUUL_CHANGES now mentions two changes, separated by a caret:
...
+ env
ZUUL_PROJECT=democlient
...
ZUUL_CHANGES=democlient:master:refs/changes/01/1/2^democlient:master:refs/changes/02/2/1
ZUUL_REF=refs/zuul/master/Z6c1cc4aea576448d9dfdcdaae46c9732
ZUUL_CHANGE_IDS=1,2 2,1
ZUUL_PIPELINE=check
ZUUL_COMMIT=d154875abac954a9ddeccfa7e875a972a3d3353b
ZUUL_PATCHSET=1
ZUUL_UUID=de438964783e41a18535f4a24b36b12e
+ zuul-cloner http://ci.localdomain:8080 democlient
...
INFO:zuul.Cloner:Prepared democlient repo with commit d154875abac954a9ddeccfa7e875a972a3d3353b
INFO:zuul.Cloner:Prepared all repositories
+ cd democlient
+ ./run_tests.sh
...
Let's have a look at the Zuul-merger democlient repository:
# git log --pretty=oneline --abbrev-commit $ZUUL_REF | |
d154875 add dummy | |
03285f3 push inital code | |
406641a Initial empty repository |
Zuul-cloner prepared the workspace in order to run the unit tests with both 03285f3 and d154875 included.
The dependent pipeline
Below is the configuration of the dependent pipeline named 'gate' in /etc/zuul/layout.yaml.
- name: gate
description: Changes that have been approved by core developers are enqueued in order in this pipeline
manager: DependentPipelineManager
precedence: normal
require:
open: True
current-patchset: True
approval:
- verified: [1, 2]
username: zuul
- code-review: 2
trigger:
gerrit:
- event: comment-added
approval:
- code-review: 2
- event: comment-added
approval:
- verified: 1
username: zuul
start:
gerrit:
verified: 0
success:
gerrit:
verified: 2
submit: true
failure:
gerrit:
verified: -2
Two elements are really important in the gate pipeline definition:
- the manager is "DependentPipelineManager"
- submit is "True"
The former tells Zuul to handle patches in that pipeline with the DependentPipelineManager logic, meaning that patches are dealt with according to the status of others patches currently in the pipeline. The latter tells Zuul to let Gerrit merge the patch on the branch if the job succeeds.
The trigger section states that a patch will enter the gate pipeline if:
- Zuul already set a score of +1 "Verified" (when the patch went through the check pipeline)
- a core reviewer set a score of +2 "Code Review"
If one of these events appears on the Gerrit stream Zuul will check that the patch also matches the "require" section. If that's the case, the jobs defined for the project under the gate pipeline will run.
Before we experiment with this pipeline be sure to define the job we want to run for demoproject within the gate pipeline. Thus update the democlient section in /etc/zuul/layout.yaml and add the gate subsection:
- name: democlient
check:
- democlient-unit-tests
gate:
- democlient-unit-tests
Validate layout.yaml and force Zuul to reload its configuration.
# zuul-server -l /etc/zuul/layout.yaml -t && kill -HUP $(pgrep zuul-server | head -1) |
Now let's merge the patch "push inital code" by setting a +2 Code Review on that pending patch. As soon as you put +2 Zuul handles the event and triggers the "democlient-unit-tests" job. The job should pass and the patch should now be merged on master.
Keep ordering
In order to verify that the gate pipeline manages to keep the validation ordering by merging working patches in the same order they appeared in the pipeline, we are going to create 2 patches:
- p1: A patch where we set a delay inside run_tests.sh (60 s)
- p2: A patch where we set a delay inside run_tests.sh (30 s)
Then we'll accept both patches with by issuing +2 Code Reviews. The order is important: p1 then p2.
As soon as we accept both patches with +2 Code Reviews, Zuul will have to handle two jobs running in the gate pipeline at the same time:
- A job with p1 applied on democlient HEAD
- A job with p1 applied on democlient HEAD then p2 applied on top of p1
On the host shell:
$ cd democlient | |
$ git checkout master && git pull | |
$ git checkout -b"p1" | |
$ sed -i 's/^SLEEP_DELAY=.*/SLEEP_DELAY=60/' run_tests.sh | |
$ git add run_tests.sh && git commit -m"p1" && git review | |
$ git checkout master && git checkout -b"p2" | |
$ sed -i '/^NT.*/a SLEEP_DELAY=30' run_tests.sh | |
$ git add run_tests.sh && git commit -m"p2" && git review |
You need to wait until both patches have been verified by the check pipeline then you can set +2 Code Review on each patch. This is important to set +2 on both patches at almost the same time, to have them in the gate pipeline concurrently. Use the commands below.
You will need both patches' from the Gerrit web UI. In my case p1's was "3,1" and p2's was "4,1".
$ ssh -p 29418 admin@ci.localdomain gerrit review 3,1 --code-review +2 | |
$ ssh -p 29418 admin@ci.localdomain gerrit review 4,1 --code-review +2 |
Now both patches are in the gate pipeline and should pass the related job, then should be merged by Zuul.
Below are the job console logs for zull cloner when testing patch p2 (4,1), where you can see that the patches currently tested in the job environment are (p1) 3,1 and (p2) 4,1.
...
ZUUL_PROJECT=democlient
...
ZUUL_CHANGES=democlient:master:refs/changes/03/3/1^democlient:master:refs/changes/04/4/1
ZUUL_REF=refs/zuul/master/Z316eaf2e04634720bfada0217695d0af
ZUUL_CHANGE_IDS=3,1 4,1
ZUUL_PIPELINE=gate
+ zuul-cloner http://ci.localdomain:8080 democlient
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/democlient-unit-tests@2
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: democlient - /var/lib/jenkins/workspace/democlient-unit-tests@2/democlient
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo democlient from upstream http://ci.localdomain:8080/democlient
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared democlient repo with commit 2da5892d57fe506ab392d026a37b660007cc783f
INFO:zuul.Cloner:Prepared all repositories
+ cd democlient
+ ./run_tests.sh
...
We can check which patches are behind $ZUUL_REF and what is important to note here is that p1 has been applied before p2 during the environment preparation by Zuul merger. Indeed as p1 has been validated before p2 and with p1 being currently in the pipeline (job not finished), Zuul assumes p1 will pass the job and will be the next patch merged on master. So instead of waiting for the end of the p1 test and its merge Zuul runs the job for p2 in parallel with p1 included. This is specific to the dependent pipeline.
# git log --pretty=oneline --abbrev-commit $ZUUL_REF | |
2da5892 Merge commit 'refs/changes/04/4/1' of ssh://ci.localdomain:29418/democlient into HEAD | |
29b7907 p2 | |
c616bd6 p1 | |
... |
Also below are the relevant log lines showing that Zuul merged both patches in the correct order. If you look carefully you'll see jobs for 3,1 (unique-id: d36a3b4f12e643faba03bbaf53bdc571) and 4,1 (unique-id: 108ef7b8a33b4b1a99c601941b0f94da) started at almost the same time but job for 4,1 finished before the other. This is expected as we set SLEEP_DELAY to 30 seconds for 4,1 and 60 seconds for 3,1. But it's important to see that Zuul kept the order by merging 3,1 before 4,1.
# grep "zuul.Gearman" /var/log/zuul/zuul.log | tail -6 | |
2015-06-15 11:37:13,527 INFO zuul.Gearman: Launch job democlient-unit-tests for change <Change 0x7febf00c26d0 3,1> with dependent changes [] | |
2015-06-15 11:37:13,547 INFO zuul.Gearman: Build <gear.Job 0x210c6d0 handle: H:172.17.0.135:79 name: build:democlient-unit-tests unique: d36a3b4f12e643faba03bbaf53bdc571> started | |
2015-06-15 11:37:18,683 INFO zuul.Gearman: Launch job democlient-unit-tests for change <Change 0x7febf00c8710 4,1> with dependent changes [<Change 0x7febf00c26d0 3,1>] | |
2015-06-15 11:37:18,699 INFO zuul.Gearman: Build <gear.Job 0x210ce50 handle: H:172.17.0.135:82 name: build:democlient-unit-tests unique: 108ef7b8a33b4b1a99c601941b0f94da> started | |
2015-06-15 11:37:57,036 INFO zuul.Gearman: Build <gear.Job 0x210ce50 handle: H:172.17.0.135:82 name: build:democlient-unit-tests unique: 108ef7b8a33b4b1a99c601941b0f94da> complete, result SUCCESS | |
2015-06-15 11:38:14,823 INFO zuul.Gearman: Build <gear.Job 0x210c6d0 handle: H:172.17.0.135:79 name: build:democlient-unit-tests unique: d36a3b4f12e643faba03bbaf53bdc571> complete, result SUCCESS | |
# grep merged /var/log/zuul/zuul.log | tail -2 | |
2015-06-15 11:38:15,031 INFO zuul.DependentPipelineManager: Reported change <Change 0x7febf00c26d0 3,1> status: all-succeeded: True, merged: True | |
2015-06-15 11:38:15,224 INFO zuul.DependentPipelineManager: Reported change <Change 0x7febf00c8710 4,1> status: all-succeeded: True, merged: True |
Discard broken patches
As previously said, Zuul manages to run jobs in parallel by assuming that patches currently in the gate pipeline will pass the tests and will be merged on the master branch. But what happens if a patch in the pipeline fails ? Let's try.
We are going to create 3 patches:
- r1: An empty file addition called "dummy1.txt"
- r2: A modification in run_tests.sh (that breaks the tests if dummy1.txt exists)
- r3: Another file addition "dummy2.txt"
$ git checkout master && git pull | |
$ git checkout -b"r1" && touch dummy1.txt && git add dummy1.txt && git commit -m"r1" && git review | |
$ git checkout master && git checkout -b"r2" && sed -i '/\#\!\/bin\/bash/a \[ -f \"dummy1.txt\" \] \&\& sleep 5 \&\& exit 1' run_tests.sh && \ | |
git add run_tests.sh && git commit -m"r2" && git review | |
$ git checkout master && git checkout -b"r3" && touch dummy2.txt && git add dummy2.txt && git commit -m"r3" && git review |
You need to wait for all patches to pass through the check pipeline and get the +1 Verified from Zuul. Then, accept them for merging by setting +2 Code Review in the correct order (to raise the bad behavior introduced in r2):
$ ssh -p 29418 admin@ci.localdomain gerrit review 5,1 --code-review +2 | |
$ ssh -p 29418 admin@ci.localdomain gerrit review 6,1 --code-review +2 | |
$ ssh -p 29418 admin@ci.localdomain gerrit review 7,1 --code-review +2 |
Below are the logs of Zuul:
# grep "zuul.Gearman" /var/log/zuul/zuul.log | tail -12 | |
2015-06-15 15:36:57,510 INFO zuul.Gearman: Launch job democlient-unit-tests for change <Change 0x210ce50 5,1> with dependent changes [] | |
2015-06-15 15:36:57,543 INFO zuul.Gearman: Build <gear.Job 0x210c8d0 handle: H:172.17.0.135:145 name: build:democlient-unit-tests unique: d7b3c50897d843f493d22f8cf2c35105> started | |
2015-06-15 15:37:03,150 INFO zuul.Gearman: Launch job democlient-unit-tests for change <Change 0x7febf00c8990 6,1> with dependent changes [<Change 0x210ce50 5,1>] | |
2015-06-15 15:37:03,173 INFO zuul.Gearman: Build <gear.Job 0x210cfd0 handle: H:172.17.0.135:148 name: build:democlient-unit-tests unique: c8801a9c2998497b87efba812c02530a> started | |
2015-06-15 15:37:09,111 INFO zuul.Gearman: Launch job democlient-unit-tests for change <Change 0x2108e90 7,1> with dependent changes [<Change 0x7febf00c8990 6,1>, <Change 0x210ce50 5,1>] | |
2015-06-15 15:37:16,202 INFO zuul.Gearman: Build <gear.Job 0x210cfd0 handle: H:172.17.0.135:148 name: build:democlient-unit-tests unique: c8801a9c2998497b87efba812c02530a> complete, result FAILURE | |
2015-06-15 15:37:16,209 INFO zuul.Gearman: Cancel build <Build bcacb34a9dcd406facc68ccf7644c4f6 of democlient-unit-tests on <Worker Unknown>> for job democlient-unit-tests | |
2015-06-15 15:37:16,252 INFO zuul.Gearman: Build <gear.Job 0x210c990 handle: H:172.17.0.135:151 name: build:democlient-unit-tests unique: bcacb34a9dcd406facc68ccf7644c4f6> started | |
2015-06-15 15:37:17,797 INFO zuul.Gearman: Launch job democlient-unit-tests for change <Change 0x2108e90 7,1> with dependent changes [<Change 0x210ce50 5,1>] | |
2015-06-15 15:37:22,263 INFO zuul.Gearman: Build <gear.Job 0x210c190 handle: H:172.17.0.135:155 name: build:democlient-unit-tests unique: 8e262d774a9d48beba4aa4d01f018f68> started | |
2015-06-15 15:37:28,865 INFO zuul.Gearman: Build <gear.Job 0x210c8d0 handle: H:172.17.0.135:145 name: build:democlient-unit-tests unique: d7b3c50897d843f493d22f8cf2c35105> complete, result SUCCESS | |
2015-06-15 15:38:07,283 INFO zuul.Gearman: Build <gear.Job 0x210c190 handle: H:172.17.0.135:155 name: build:democlient-unit-tests unique: 8e262d774a9d48beba4aa4d01f018f68> complete, result SUCCESS | |
# grep merged /var/log/zuul/zuul.log | tail -3 | |
2015-06-15 15:37:29,051 INFO zuul.DependentPipelineManager: Reported change <Change 0x210ce50 5,1> status: all-succeeded: True, merged: True | |
2015-06-15 15:37:29,212 INFO zuul.DependentPipelineManager: Reported change <Change 0x7febf00c8990 6,1> status: all-succeeded: False, merged: False | |
2015-06-15 15:38:07,479 INFO zuul.DependentPipelineManager: Reported change <Change 0x2108e90 7,1> status: all-succeeded: True, merged: True |
Four jobs have been started by Zuul :
- HEAD + r1
- HEAD + r1 + r2 (Failed)
- HEAD + r1 + r2 + r3 (Canceled)
- HEAD + r1 + r3
From the logs we clearly see that zuul began to start the job to verify r3 (7,1) with r2 (6,1) and r3 (5,1) applied on top of master, but r2 already failed so Zuul asks to cancel the job for r3 as something went wrong with r2. Zuul then started another job with r1 and r3 applied on top of master, bypassing the faulty patch r2.
If you have a look to the Gerrit WEB UI you'll see that only r1 and r3 have been merged and r2 has been rejected "-2 Verified". It's up to the author to fix and submit a new patchset.
Four jobs have been started; that also means Zuul-merger created four $ZUUL_REF.
So thanks to the gate pipeline we haven't merged r2 on the master branch. Note that r2 passed the test in the check pipeline because r1 was not put inside the test environment.
In a context where more than one person are allowed to accept a patch and also where jobs can take a long time to run I let you imagine how this behavior can save time. It minimizes the need to revert commits to fix the master branch.
Cross projects jobs
Let's say you have two projects repositories on Gerrit and both interact together. For instance this can be the case for:
- a client and a server
- a software and its plugins
Also you have a test "like a functional test" that verifies that a plugin interacts well with the software. You want to trigger the functional test when a patch is proposed either on the software or on the plugin.
- If a patch is proposed on the software then you want to run the functional test with the patch applied on software master HEAD, and using the plugin master HEAD.
- If a patch is proposed on the plugin you want to run the functional test with the patch applied on plugin master HEAD, and using the software master HEAD.
Again Zuul-cloner can be used to prepare the functional test's job environment thanks to the git repositories prepared by zuul-merger.
We are going to test that with a second project called "demolib" that is a dummy library designed to run with "democlient".
First create the demolib project on Gerrit and setup Zuul/Jenkins for testing this new project.
- Setup Zuul/Jenkins
- Create a project called "demolib" using the Admin account on Gerrit. (be sure to check "create an empty commit")
In /etc/jenkins_jobs/jobs/jjb.yaml
- job:
name: demolib-unit-tests
defaults: global
builders:
- shell: |
env | grep ZUUL
zuul-cloner http://ci.localdomain:8080 $ZUUL_PROJECT
cd $ZUUL_PROJECT
./run_tests.sh
- job:
name: demo-functional-tests
defaults: global
builders:
- shell: |
env | grep ZUUL
zuul-cloner http://ci.localdomain:8080 democlient
zuul-cloner http://ci.localdomain:8080 demolib
cd democlient
DEMOLIBPATH=../demolib ./run_functional-tests.sh
- project:
name: democlient
node: master
jobs:
- democlient-unit-tests
- demo-functional-tests
- project:
name: demolib
node: master
jobs:
- demolib-unit-tests
- demo-functional-tests
Above we added a job to run the unit tests for the project demolib.
Also we added a job "demo-functional-tests" that will run the functional tests of democlient. democlient needs demolib to behave as expected.
The demo-functional-tests job uses zuul-cloner in order to fetch democlient and demolib inside the workspace, then starts run_functional-tests.sh by setting the path to the demolib codebase in the job's workspace.
We also need to configure Zuul to associate additional jobs with the related projects. Thus modify /etc/zuul/layout.yaml like below:
projects:
- name: democlient
check:
- democlient-unit-tests
- demo-functional-tests
gate:
- democlient-unit-tests
- demo-functional-tests
- name: demolib
check:
- demolib-unit-tests
- demo-functional-tests
gate:
- demolib-unit-tests
- demo-functional-tests
Here we configure Zuul to start "demolib-unit-tests" inside the pipelines check and gate for the demolib project. We also ask to run "demo-functional-tests" for both projects as well for the check and gate pipelines.
Run Jenkins job builder and restart Zuul:
# jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini update /etc/jenkins_jobs/jobs | |
# zuul-server -l /etc/zuul/layout.yaml -t && kill -HUP $(pgrep zuul-server | head -1) |
And push the initial code on "demolib": (commands to be perform from your laptop)
$ git clone https://github.com/morucci/demolib.git demolib_origin | |
$ git clone http://ci.localdomain:8080/demolib | |
$ cd demolib | |
$ git checkout -b"initial_code" | |
$ cp -R ../demolib_origin/* . | |
$ cat > .gitreview << EOF | |
[gerrit] | |
host=ci.localdomain | |
port=29418 | |
project=demolib | |
EOF | |
$ git config --add gitreview.username "admin" | |
$ git review -s | |
$ git add .gitreview * | |
$ git commit -m"push inital code" | |
$ git review |
By looking at the job console logs of "demo-functional-tests" you can see an output like this:
...
ZUUL_PROJECT=demolib
ZUUL_BRANCH=master
ZUUL_URL=http://ci.localdomain/p
ZUUL_CHANGE=8
ZUUL_CHANGES=demolib:master:refs/changes/08/8/1
ZUUL_REF=refs/zuul/master/Z0d55ecb440c1415ca61cf36ad690e2cd
ZUUL_CHANGE_IDS=8,1
ZUUL_PIPELINE=check
ZUUL_COMMIT=e8c81dd33ece1b0f651a143aff9f0e6a2df2bddd
ZUUL_PATCHSET=1
ZUUL_UUID=84bef62dfb0a44f1bf62df631aacd7d2
+ zuul-cloner http://ci.localdomain:8080 democlient
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/demo-functional-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: democlient - /var/lib/jenkins/workspace/demo-functional-tests/democlient
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo democlient from upstream http://ci.localdomain:8080/democlient
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Falling back to branch master
INFO:zuul.Cloner:Prepared democlient repo with branch master
INFO:zuul.Cloner:Prepared all repositories
+ zuul-cloner http://ci.localdomain:8080 demolib
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/demo-functional-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: demolib - /var/lib/jenkins/workspace/demo-functional-tests/demolib
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo demolib from upstream http://ci.localdomain:8080/demolib
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared demolib repo with commit e8c81dd33ece1b0f651a143aff9f0e6a2df2bddd
INFO:zuul.Cloner:Prepared all repositories
+ cd democlient
+ DEMOLIBPATH=../demolib
+ ./run_functional-tests.sh
...
Below is the state of $ZUUL_REF on democlient and demolib:
# cd /var/lib/zuul/git/democlient/ | |
# git log --pretty=oneline --abbrev-commit $ZUUL_REF | |
fatal: ambiguous argument 'refs/zuul/master/Z0d55ecb440c1415ca61cf36ad690e2cd': unknown revision or path not in the working tree. | |
... | |
# cd /var/lib/zuul/git/demolib | |
# git log --pretty=oneline --abbrev-commit $ZUUL_REF | |
10f615e push inital code | |
3b2370d Initial empty repository |
Here the functional test job via zuul-cloner has prepared the job environment by setting demolib to the proper commit fetched from $ZUUL_REF. For democlient zuul-cloner fell back to the master branch as the merger has not prepared a temporary branch under $ZUUL_REF, since it wasn't needed.
But in some circumstances you'll have a branch referenced by $ZUUL_REF for multiple projects:
- The patch has been accepted then passes through the dependent pipeline "gate" but another patch, on democlient, has been accepted before and is also in the gate pipeline in front of the demolib patch.
- The commit message of the demolib patch indicates one or a couple of dependencies via the keyword "Depends-on".
You can approve the patch on demolib by setting +2 CR. The patch passes through the gate pipeline and is merged.
Now we will add two more patches and manage to have them go through the dependent pipeline gate at the same time:
Let's create a patch on democlient and on demolib:
$ cd democlient | |
$ git checkout master && git pull | |
$ git checkout -b"c1" | |
$ sed -i 's/^SLEEP_DELAY=.*/SLEEP_DELAY=60/' run_tests.sh | |
$ git add run_tests.sh && git commit -m"c1" && git review | |
$ cd demolib | |
$ git checkout master && git pull | |
$ git checkout -b"d1" | |
$ touch dummy1.txt | |
$ git add dummy1.txt && git commit -m"d1" && git review |
Then we accept both patches on democlient and demolib. c1 then d1:
$ ssh -p 29418 admin@ci.localdomain gerrit review 9,1 --code-review +2 | |
$ ssh -p 29418 admin@ci.localdomain gerrit review 10,1 --code-review +2 |
Below are the job console logs of demo-functional-tests in the gate pipeline for the patch d1 against demolib, where you can see that zuul-cloner fetched the tip of $ZUUL_REF for both projects. Indeed as demolib and democlient share at least one test with the same name "demo-functional-test", Zuul built a shared queue between demolib and democlient and created the same $ZUUL_REF on both repositories.
The behavior of the gate pipeline explained above (in the previous chapter) is applied on two or more project's repositories; and so patch d1 (on demolib) has been validated via the functional test with patch c1 applied on democlient.
...
ZUUL_PROJECT=demolib
ZUUL_CHANGES=democlient:master:refs/changes/03/3/1^demolib:master:refs/changes/04/4/1
ZUUL_REF=refs/zuul/master/Ze6d383a9614041f9a4e43879996fd1f2
ZUUL_CHANGE_IDS=9,1 10,1
...
+ zuul-cloner http://ci.localdomain:8080 democlient
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/demo-functional-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: democlient - /var/lib/jenkins/workspace/demo-functional-tests/democlient
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo democlient from upstream http://ci.localdomain:8080/democlient
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared democlient repo with commit 1e2150e51d25505a8a8f26f4c23d1a4fc136ee91
INFO:zuul.Cloner:Prepared all repositories
+ zuul-cloner http://ci.localdomain:8080 demolib
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/demo-functional-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: demolib - /var/lib/jenkins/workspace/demo-functional-tests/demolib
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo demolib from upstream http://ci.localdomain:8080/demolib
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared demolib repo with commit 600e36667c7850cb856e1281baad502eb05aa7c2
INFO:zuul.Cloner:Prepared all repositories
...
We can verify which commits were under $ZUUL_REF:
# cd /var/lib/zuul/git/democlient/ | |
# git log --pretty=oneline --abbrev-commit refs/zuul/master/Ze6d383a9614041f9a4e43879996fd1f2 | |
1e2150e c1 | |
4c39035 push inital code | |
f3cc73e Initial empty repository | |
# cd /var/lib/zuul/git/demolib | |
# git log --pretty=oneline --abbrev-commit refs/zuul/master/Ze6d383a9614041f9a4e43879996fd1f2 | |
600e366 d1 | |
10f615e push inital code | |
3b2370d Initial empty repository |
Depends-on
Zuul can detect one or multiple mentions of the folowing string in a commit message:
"Depends-on: <changeid>"
When a patch needs another one on the same project as its base it is easy to just make the patch dependent on a specific parent SHA commit id but when it depends on a patch (not yet merged) from another project, then you need to provide some extra information via "Depends-on".
For instance, let's assume you want to add a new function to demolib:
$ cd demolib | |
$ git checkout master && git pull | |
$ git checkout -b"f1" | |
$ touch new_function.py | |
$ git add new_function.py && git commit -m"Add new function" && git review |
Retrieve the Change-Id from your last commit message. Thanks to it you'll be able to declare this patch as a dependency in another patch.
$ CHANGEID=$(git show --quiet HEAD | sed -n '/Change-Id/ s/.*Change-Id: //p') |
Use the "Depends-On" keyword with the Change-Id to indicate that democlient needs that new function in order to implement a new feature.
$ cd democlient | |
$ git checkout master && git pull | |
$ git checkout -b"g1" | |
$ touch new_feature.py | |
$ git add new_feature.py && echo -e "Add new feature\n\nDepends-On: $CHANGEID" | git commit --file - | |
$ git review |
Below are the logs of demo-functional-test in the check pipeline for our patch g1 on democlient.
...
ZUUL_PROJECT=democlient
ZUUL_CHANGES=demolib:master:refs/changes/11/11/1^democlient:master:refs/changes/12/12/1
ZUUL_REF=refs/zuul/master/Z18f19080f4c94d4c85bf689ad58588a7
ZUUL_CHANGE_IDS=11,1 12,1
ZUUL_PIPELINE=check
...
+ zuul-cloner http://ci.localdomain:8080 democlient
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/demo-functional-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: democlient - /var/lib/jenkins/workspace/demo-functional-tests/democlient
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo democlient from upstream http://ci.localdomain:8080/democlient
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared democlient repo with commit c7aeebda47e0548ed7b7dfa4fb43c660c532ee7a
INFO:zuul.Cloner:Prepared all repositories
+ zuul-cloner http://ci.localdomain:8080 demolib
INFO:zuul.CloneMapper:Workspace path set to: /var/lib/jenkins/workspace/demo-functional-tests
INFO:zuul.CloneMapper:Mapping projects to workspace...
INFO:zuul.CloneMapper: demolib - /var/lib/jenkins/workspace/demo-functional-tests/demolib
INFO:zuul.CloneMapper:Expansion completed.
INFO:zuul.Cloner:Preparing 1 repositories
INFO:zuul.Cloner:Creating repo demolib from upstream http://ci.localdomain:8080/demolib
INFO:zuul.Cloner:upstream repo has branch master
INFO:zuul.Cloner:Prepared demolib repo with commit c6bfc270799e522fe40a95125c8810f0176998bc
INFO:zuul.Cloner:Prepared all repositories
...
If we look at the tip of $ZUUL_REF branch on the democlient merger repository we can see the patch (Add new function) we put as a dependency for our patch (Add new feature):
# cd /var/lib/zuul/git/demolib | |
# git log --pretty=oneline --abbrev-commit refs/zuul/master/Z18f19080f4c94d4c85bf689ad58588a7 | |
99217a4 Add new function | |
44ecf87 d1 | |
e8c81dd push inital code | |
c4b652d Initial empty repository | |
# cd /var/lib/zuul/git/democlient | |
# git log --pretty=oneline --abbrev-commit refs/zuul/master/Z18f19080f4c94d4c85bf689ad58588a7 | |
c7aeebd Add new feature | |
1e2150e c1 | |
4c39035 push inital code | |
f3cc73e Initial empty repository |
The dependent pipeline "gate" supports "Depends-On" as well.
The "Depends-On" can also be useful to ensure a patch cannot be merged if a cross project dependency has not been merged before, or if it is not above in the gate pipeline. For instance here are the Zuul logs if you intend to set +2 CR on "c7aeebd Add new feature" before "Add new function" is merged.
2015-06-17 13:23:19,311 INFO zuul.Scheduler: Adding democlient, <Change 0x7f634401e250 12,1> to <Pipeline gate> | |
2015-06-17 13:23:19,311 DEBUG zuul.DependentPipelineManager: Considering adding change <Change 0x7f634401e250 12,1> | |
2015-06-17 13:23:19,312 DEBUG zuul.DependentPipelineManager: Checking for changes needed by <Change 0x7f634401e250 12,1>: | |
2015-06-17 13:23:19,312 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f634401e250 12,1> needs change <Change 0x7f634402ef50 11,1>: | |
2015-06-17 13:23:19,312 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f634402ef50 11,1> is needed but can not be merged | |
2015-06-17 13:23:19,313 DEBUG zuul.DependentPipelineManager: Failed to enqueue changes ahead of <Change 0x7f634401e250 12,1> |
So how could you use Zuul in your company ?
Here are some use cases where Zuul can be useful:
- You already have Gerrit and Jenkins as a CI platform and you want to perform automatic gating with Zuul (dependent pipeline). In this case Zuul should be easy to integrate in such platform.
- Your software is based on OpenStack components and you want to be confident that the next patches that will land on the master branch won't break your software. Zuul can be used to build a "third party CI" and react to reviews.openstack.org events.
- You want to implement a Continuous Integration/Continuous Deployment workflow for your software; then you can bind the required jobs on pipelines such as check, gate (for Continuous Integration and ensuring code quality) and post (for Continuous Deployment of master's HEAD in production).
For more information about Zuul have a look at the official documentation.
If you want to see Zuul in action jump here.
As we think Zuul is a really interesting component to have in software development workflow we have integrated it in an open source platform CI/CD we develop called Software Factory. You can check out what we do here.
And see it in action here.
[…] Dive into Zuul – Gated commit system […]
Excellent blog post. Clear explanations and demo. Thanks a lot.
(there is a bug introduced in python-jenkins 0.4.9 which affects jenkins-job builder. If you want to closely follow/reproduce the instructions in the blog post, install python-jenkins 0.4.8 or have a loot at https://bugs.launchpad.net/python-jenkins/+bug/1501441)
Hi
Its a useful blog.. When i tried to setup this on my system. I am facing some issue in bringing jenkins service up …It is throwing service unavailable error..
Could someone help me out..
Hi,
If we have two tickets for one feature and those two tickets should be verified together and merge at the same time,how can gate or depends-on help us?
In your example, one ticket maybe go into master first, it is not allowed in my case.