This post is intended to walk somone through the process of establishing an external testing platform that is linked with the upstream OpenStack continuous integration platform. If you haven’t already, please do read the first article in this series that discusses the upstream OpenStack CI platform in detail. At the end of the article, you should have all the background information on the tools needed to establish your own linked external testing platform.

What Does an External Test Platform Do?

In short, an external testing platform enables third parties to run tests — ostensibly against an OpenStack environment that is configured with that third party’s drivers or hardware — and report the results of those tests on the code review of a proposed patch. It’s easy to see the benefit of this real-time feedback by taking a look at a code review that shows a number of these external platforms providing feedback. In this screenshot, you can see a number Verified +1 and one Verified -1 labels added by external Neutron vendor test platforms on a proposed patch to Neutron:

Verified +1 and -1 labels added by external testing systems on a Neutron patch

本文介绍如何构建与 OpenStack continuous integration platform贯穿的测试平台。开始前最好对本文upstream OpenStack CI platform in detail中的背景有所了解. 读完本文后,你将具备构建测试平台所需的所有背景知识。


简单的说,就是 运行第三方的独立测试 — 通过将三方驱动或硬件配置到 OpenStack 环境中 — 然后在代码审查的过程中展示相关的测试报告. 通过这个实时的代码审查回馈能很容易的把控代码质量. 下图中,你会看到Neutron的外部测试平台加入的numberVerified +1 and one Verified -1 标签 :

Verified +1 and -1 labels added by external testing systems on a Neutron patch

Verified +1 and -1 labels added by external testing systems on a Neutron patch

Each of these systems, when adding a Verified label to a review does so by adding a comment to the review. These comments contain links to artifacts from the external testing system’s test run for this proposed patch, as shown here:

Comments added to a review by the vendor testing platforms

Comments added to a review by the vendor testing platforms

The developer submitting a patch can use those links to investigate why their patch has caused test failures to occur for that external test platform.

测试平台引入的Verified +1 and -1 标签

每添加一个新的标签就会产生一条comments. 点击注释就会链接到测试平台上的相关测试:

Comments added to a review by the vendor testing platforms

Comments 对应着一次测试平台的review


Why Set Up an External Test Platform?

The benefits of external testing integration with upstream code review are numerous:

  • A tight feedback loop

  • The third party gets quick notifications that a proposed patch to the upstream code has caused a failure in their driver or configuration. The tighter the “feedback loop”, the faster fixes can be identified

  • Better code coverage

  • Drivers and plugins that may not be used in the default configuration for a project can be tested with the same rigor and frequency as drivers that are enabled in the upstream devstack VM gate tests. This prevents bitrot and encourages developers to maintain code that is housed in the main source trees.

  • Increased consistency and standards

  • Determining a standard set of tests that prove a driver implements the full or partial API of a project means that drivers can be verified to work with a particular release of OpenStack. If you’ve ever had a conversation with a potential deployer of OpenStack who wonders how they know that their choice of storage or networking vendor, or underlying hypervisor, actually works with the version of OpenStack they plan to deploy, then you know why this is a critical thing!



  • 及时获得反馈

  • 测试平台能提早发现相关补丁提交后可能造成的失败,这能缩短问题解决时间

  • 更好的代码覆盖率

  • 通过模拟测试插件和驱动程序来减少对真实测试环境的依赖。以此为开发人员提供更多的时间专注于业务代码的质量

  • 持续提高质量标准

  • 通过相关用例来测试部分或者全部的驱动程序API以保证与OpenStack的兼容性. 如果你和OpenStack的开发人员聊过,就会知道他们面对如何选择存储或网络供应商或是后台管理模式去兼容将要部署的 OpenStack问题时,这一优势会多么明显!

Why might you be thinking about how to set up an external testing platform? Well, a number of OpenStack projects have had discussions already about requirements for vendors to complete integration of their testing platforms with the upstream OpenStack CI platform. The Neutron developer community is ahead of the game, with more than half a dozen vendors already providing linked testing that appears on Neutron code reviews.

The Cinder project also has had discussions around enforcing a policy that any driver that is in the Cinder source tree have tests run on each commit to validate the driver is working properly. Similarly, the Nova community has discussed the same policy for hypervisor drivers in that project’s source tree. So, while this may be old news for some teams, hopefully this post will help vendors that are new to the OpenStack contribution world get integrated quickly and smoothly.

何必再去思考如何选择测试平台的问题? 已经大把OpenStack项目和供应商探讨过集成测试平台到上游的OpenStack持续集成平台的需求了.Neutron开发社区已经走在前面 ahead of the game,大约快一打的供应商已经在Neutron代码审查中加入了测试的链接.

 Cinder项目在讨论 discussions 强制推行一有代码提交就测试驱动的正确性策略。 同样的,Nova 社区也讨论了 discussed 监控源码库的相关策略. 也许这对于有些团队来说不是什么新鲜事, 带希望本文能给新接触 OpenStack的供应商们提供更快融入的帮助 

The Tools You Will Need

The components involved in building a simple linked external testing system that can listen to and notify the upstream OpenStack continuous integration platform are as follows:

  • Jenkins CI

  • The server that is responsible for executing jobs that run tests for a project

  • Zuul

  • A system that configures and manages event pipelines that launch Jenkins jobs

  • Jenkins Job Builder (JJB)

  • Makes construction/maintenance of Jenkins job config XML files a breeze

  • Devstack-Gate and Nodepool Scripts

  • A collection of scripts that constructs an OpenStack environment from source checkouts

I’ll be covering how to install and configure the above components to build your own testing platform using a set of scripts and Puppet modules. Of course, there are a number of ways that you can install and configure any of these components.


能够和OpenStack持续集成平台协同的测试平台组件如下 :

  • Jenkins CI

  • 运行相关项目测试任务的服务器

  • Zuul

  • 组织运行jenkins任务机制的系统

  • Jenkins Job Builder (JJB)

  • 简化Jenkins任务配置文件的创建\维护工作

  • Devstack-Gate and Nodepool 脚本

  • 从源码库构建OpenStack 环境的脚本

下面我会说明如何使用脚本和Puppet实现上述测试平台的组件功能. 当然还有其他的方法实现相同的功能。

You can manually install it somewhere by following the install instructions in the component’s documentation. However, I do not recommend that. The problem with manual installation and configuration is two-fold:

  1. If something goes wrong, you have to re-install everything from scratch. If you haven’t backed up your configuration somewhere, you will have to re-configure everything from memory.

  2. You cannot launch a new configuration or instance of your testing platform easily, since you have to manually set everything up again.

A better solution is to use a configuration management system, such as Puppet, Chef, Ansible or SaltStack to manage the deployment of these components, along with a Git repository to store configuration data. In this article, I will show you how to install an external testing system on multiple hosts or virtual machines using a set of Bash scripts and Puppet modules I have collected into a source repository on GitHub. If you don’t like Puppet or would just prefer to use a different configuration management tool, that’s totally fine. You can look at the Puppet modules in this repo for inspiration (and eventually I will write some Ansible scripts in the OpenStack External Testing project, too).

可以根据文档手工安装相关组件. 但我不建议这么做,手动安装有利有弊:

  1. 一旦有地方出错,所有工作全部需要返工。如果之前做了什么配置修改,现在全凭记忆来恢复。

  2. 无法方便的建立新的测试平台示例.如果要实现就得全部从新配置

好的办法就是使用配置管理平台, 比如 Puppet, Chef, Ansible or SaltStack 来管理组件的部署, 同时使用Git管理配置库. 本文将 通过Bash脚本和Puppet modules实现在多个host或是虚拟机上构建测试平台。相关代码在 source repository on GitHub. 如果你不想用 Puppet或是想用其他的工具, 也没啥问题. 从刚才的源码库中也能得到不少启发 (以后我还会写一些OpenStack 外部测试项目的脚本).


Before I go into the installation instructions, you will need to take care of a few things. Follow these detailed steps and you should be all good.

Getting an Upstream Service Account

In order for your testing platform to post review comments to Gerrit code reviews on openstack.org, you will need to have a service account registered with the OpenStack Infra team. See this link for instructions on getting this account.

In short, you will need to email the OpenStack Infra mailing list an email that includes:

  • The email address to use for the system account (must be different from any other Gerrit account)

  • A short account username that will appear on code reviews

  • (optional) A longer account name or description

  • (optional but encouraged) Include your contact information (IRC handle, your email address, and maybe an alternate contact’s email address) to assist the upstream infrastructure team

  • The public key for an SSH key pair that the service account will use for Gerrit access. Please note that there should be no newlines in the SSH key

Don’t have an SSH key pair for your Gerrit service account? You can create one like so:

ssh-keygen -t rsa -b 1024 -N '' -f gerrit_key

The above will produce the key pair: a pair of files called gerrit_key and gerrit_key.pub. Copy the text of the gerrit_key.pub into the email you send to the OpenStack Infra mailing list. Keep both the files handy for use in the next step.




为了能够发布测试平台的评审备注到 openstack.org, 需要先从 OpenStack Infra 团队注册一个账户. 详细如下 this link for instructions 

简单来说需要向 OpenStack Infra mailing list 发送一份邮件,里面包含下述内容:

  • 邮件地址作为系统账户(必须是唯一的Gerrit账户)

  • 账户名缩写会显示在代码评审里

  • (可选) 账户详细描述

  • (可选但推荐) 联系方式 (IRC handle, 邮箱地址或是其它间接联系邮箱) 方便上游基础团队联系

  • 访问Gerrit的SSH键值组公匙 . 注意SSH 中不要多出新代码行

还没有Gerrit服务账户的的SSH键值组? 可以创建一个:

ssh-keygen -t rsa -b 1024 -N '' -f gerrit_key

上面命令会生成一个兼职组: 被命名为gerrit_keyandgerrit_key.pub的文件. 将gerrit_key.pub中的内容通过邮件发送到 OpenStack Infra mailing list. 保留上述两个东西方便下面使用。

Create a Git Repository to Store Configuration Data

When we install our external testing platform, the Puppet modules are fed a set of configuration options and files that are specific to your environment, including the SSH private key for the Gerrit service account. You will need a place to store this private configuration data, and the ideal place is a Git repository, since additions and changes to this data will be tracked just like changes to source code.

I created a source repository on GitHub that you can use as an example. Instead of forking the repository, like you might would normally do, I recommend instead just git clone’ing the repository to some local directory, and making it your own data repository:

git clone git@github.com:jaypipes/os-ext-testing-data ~/mydatarepo
cd mydatarepo
rm -rf .git
git init .
git add .
git commit -a -m "My new data repository"

Now you’ve got your own data repository to store your private configuration data and you can put it up in some private location somewhere — perhaps in a private organization in GitHub, perhaps on a Git server you have somewhere.


开始安装测试平台后Puppet modules会保存一系列系统的配置文件,包括Gerrit服务账户的SSH私匙。保存这些文件最理想的东西就是Git配置库了, 任何的变动都会被保存下来就想保存源码那样。

我开了一个账户 source repository on GitHub 作为示例. 和以往不同的是,我建议使用git clone代码到本地库使用,而不要使用fork的模式:

git clone git@github.com:jaypipes/os-ext-testing-data ~/mydatarepo
cd mydatarepo
rm -rf .git
git init .
git add .
git commit -a -m "My new data repository"

现在就得到了一个本地的配置库来保存相关的配置文件,随便放在哪里都没问题 — 也许是 GitHub,甚至其它地方的Git服务器上.

Put Your Gerrit Service Account Private Key Into the Data Repository

The next thing you will want to do is add your SSH key pair to the repository that you used in the step above that had you register an upstream Gerrit service account.

If you created a new key pair using the ssh-keygen command above. You would copy the gerrit_key file into your data repository.

If you did not create a new key pair (you used an existing key pair) or you created a key pair that wasn’t called gerrit_key, simply copy that key pair into the data repository, then open up the file called vars.sh, and change the following line in it:


And change gerrit_key to the name of your SSH private key.


现在要把上一步创建的访问Gerri 服务账户的键值组也放到仓库里。

如果用ssh-keygen命令创建了新的键值组. 也需要将gerrit_key文件拷贝到仓库里.

如果既没有创建新的键值组 (用的原来的) 或者名字不是 gerrit_key, 那就拷贝该文件到仓库再打开 vars.sh, 修改下述代码:


将gerrit_key 改成对应的SSH私匙名.

Set Your Gerrit Account Username

Next, open up the file vars.sh in your data repository (if you haven’t already), and change the following line in it:

export UPSTREAM_GERRIT_USER=jaypipes-testing

And replace jaypipes-testing with your Gerrit service account username.

Set Your Vendor Name in the Test Jenkins Job

Next, open up the file etc/jenkins_jobs/config/projects.yaml in your data repository. Change the following line in it:

  vendor: myvendor

Change myvendor to your organization’s name.

(Optional) Create Your Own Jenkins SSH Key Pair

I have a private/public SSH key pair (named jenkins_key[.pub] in the example data repository. Due to the fact that I’ve put the private key in there, it’s no longer useful as anything other than an example, so you may want to recreate your own. Do so like so:

ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key
git commit -a -m "Changed jenkins key to a new private one"

Save Changes in Your Data Repository

OK, we’re done with the first changes to your data repository and we’re ready to install a Jenkins master node. But first, save your changes and push your commit to wherever you are storing your data repository privately:

git add .
git commit -a -m "Added Gerrit SSH key and username"
git push

设置Gerrit 账户用户名

下面打开创库里的filevars.sh (假定你还没打开), 修改下面代码:

export UPSTREAM_GERRIT_USER=jaypipes-testing


在Test Jenkins Job上设置供应商名称

打开 etc/jenkins_jobs/config/projects.yaml . 修改下面代码:

  vendor: myvendor


(可选) 创建 Jenkins SSH 键值对

在数据仓中我有公有/私有的SSH键值对 (叫做jenkins_key[.pub]. 因为已经把私匙放过去了现在其实也没啥用,就当个例子吧。如果想建个新的,这样操作:

ssh-keygen -t rsa -b 1024 -N '' -f jenkins_key
git commit -a -m "Changed jenkins key to a new private one"


好的,现在已经处理好了数据仓库的配置可以开始安装Jenkins master服务器了. 但记住先保存一下相关的更改放到配置库里:

git add .
git commit -a -m "Added Gerrit SSH key and username"
git push