最近在部署cloudfoundry测试环境,遇到的问题不少,不停地在google group、github以及cf官方之间穿梭,抽空总结出来,分享的同时让大家少走弯路。利用BOSH部署cf-release有几个步骤,BOSH-cli-->MIcroBosh-->Bosh-->cf-release,先安装 bosh-cli,通过bosh-cli部署Miro bosh,再通过Micro bosh部署 bosh集群,最后利用 bosh集群来部署cf-realse集群。这些相信大家都很了解了,一些背景知识本文不打算涉及,只专注于部署过程本身。
本文先详细讲解如何部署MicroBosh,迈出第一步。
注:基于openstack IAAS,版本Folsom 。
第一篇 :部署MicroBOSH
一、BOSH 整体架构
官方的一张图
发现其实BOSH的架构与CloudFoundry几乎一样。
outer Shell部分
CLI=VMC ,指令入口
director=CloudController,控制器,接收CLI传成的指令,执行并返回
DB = CCDB ,存储着系统的一些数据
Blobstore = Blobstore or NFS,存储着package,jobs 以及uncompiled package,stemcell
MessageBus=NATS,消息总线,本身就是用NATS。与CF一样
HealtehMonitor = HealthManager ,健康检查,接收每个vm通过agent传回的数据。
Inner Shell = DEA ,vms就形如droplets
与CF相比,这里多了一个Worker,实际上可以理解为线程,诸如编译bosh-release,cf-release的工作,以及打包Jobs,都得另外启动虚拟机来完成的,这个虚拟机就是Worker,当然也是director通过 openstack CPI来创建的。在测试环境中,为了节省资源,将worker 数据设置为1就行。
CF架构中,droplets是由开发者开发上传的,而对应到bosh的vm,是由openstack创建的。BOSH与CF的架构几乎一致。因为都是VMware主导,这就不奇怪了。
二、环境要求
组件 | 节点数目 | 操作系统 | 可否是物理机 |
---|---|---|---|
BOSH CLI | 1 | Ubuntu | 可以 |
Micro BOSH | 1 | Ubuntu | 不可以 |
BOSH | 6 | Ubuntu | 不可以 |
Cloud Foundry | 20+ | Ubuntu | 不可以 |
合计: | 30- |
cloudfoundry 在测试环境,很多service是可选的。这个就可以少很多。具体的再定。还有隐藏的Worker,它完成任务就删掉,所有还得准备些冗余。
三、准备虚拟机镜像
CloudFoundry运行在ubuntu环境,虽然理论上其他Linux版本也可以。先准备一个ubuntu操作系统镜像,用于创建部署过程中需要的虚拟机,以及自定义stemcell模板的制作,推荐使用ubuntu 10.04 LTS版本,在openstack dashboard 上传一个镜像。
本文用的是openstack官方上的12.04版本。
四、创建BOSH-cli
有了系统镜像之后,就可以创建BOSH-cil虚拟机了。
4.1 用admin帐号创建一个flavor,测试环境,虚拟机要求不太高:1个vcpu,10G root disk, 10G ephemeral disk,1024M RAM就OK了。
4.2 回到租户帐号,创建密钥对test.pem,浏览器会自动下载到本地,请妥善保存。
4.3 创建虚拟机,在Access&Security 中选择密钥对 test,再创建。
4.4 登录虚拟机, 从本机登录,要安装ssh。
ssh -i test.pem ubuntu@ip
五、初始化 BOSH-cli环境
利用bosh-ci来部署micro bosh, 我们需要ruby环境,bosh-cli,bosh-agent ,bosh-deployer等。
5.1 更换软件源
这步可选,由于国内的网络因素,普遍认为163源比较靠谱。
cp /etc/apt/sources.list /etc/apt/sources.ilst.bak vim /etc/apt/sources.list
用以下源替代
deb http://mirrors.163.com/ubuntu/ precise main universe restricted multiverse deb-src http://mirrors.163.com/ubuntu/ precise main universe restricted multiverse deb http://mirrors.163.com/ubuntu/ precise-security universe main multiverse restricted deb-src http://mirrors.163.com/ubuntu/ precise-security universe main multiverse restricted deb http://mirrors.163.com/ubuntu/ precise-updates universe main multiverse restricted deb http://mirrors.163.com/ubuntu/ precise-proposed universe main multiverse restricted deb-src http://mirrors.163.com/ubuntu/ precise-proposed universe main multiverse restricted deb http://mirrors.163.com/ubuntu/ precise-backports universe main multiverse restricted deb-src http://mirrors.163.com/ubuntu/ precise-backports universe main multiverse restricted deb-src http://mirrors.163.com/ubuntu/ precise-updates universe main multiverse restricted更新软件源
apt-get update
5.2 安装ruby环境
方法一:
有一个现成的shell脚本,跑下就行了。脚本安装了ruby,以及一些必要的软件包,还有bosh源码,并编译。
sudo su - export ORIGUSER=ubuntu curl -s https://raw.github.com/KingJames3/bosh-getting-started/master/scripts/create-bosh-cli.sh | bash source /etc/profile
安装ruby时会比较慢,也是跟国内网络环境有关。如果觉得太慢,可以查看prepare_inception_openstack.sh脚本,一步步执行,眼见为实。安装到ruby时,如遇到
网络问题,建议改成淘宝的gem镜像,再执行一遍。最后一步的git clone bosh-release暂时不需要,过程也比较慢。这个shell也是为自定义stemcell准备的,
由于我们用现成的micro-bosh-stemcell,里面包含了bosh-release。所以建议把prepare_inception_openstack.sh下载下来,删除agent,openstack-
registry, bosh-release那几段,再执行。
gem source -r http://rubygems.org/
gem source -a http://ruby.taobao.org
环境安装完后,确认下。
ubuntu@bosh-cli:~/stemcell$ bosh -v BOSH 1.0.3 ubuntu@bosh-cli:~/stemcell$ bosh help micro micro show micro bosh sub-commands micro agent <args> Send agent messages micro apply <spec> Apply spec micro delete Delete micro BOSH instance (including persistent disk) micro deploy [<stemcell>] [--update] Deploy a micro BOSH instance to the currently selected deployment --update update existing instance micro deployment [<name>] Choose micro deployment to work with, or display current deployment micro deployments Show the list of deployments micro status Display micro BOSH deployment status
没问题后就可以开始干活了。
方法二:
bosh cli 就是需要有ruby环境,安装bosh_cli和bosh_deployer就行了。所以如果不嫌麻烦,可以自己一步步来。假设创建完ubuntu vm
1. 安装ruby环境
curl -L get.rvm.io | bash -s stable source /etc/profile.d/rvm.sh rvm install 1.9.3-p374 # 这里用p374,新版的p392在安装nokogiri时有问题,没有深入排查 rvm 1.9.3 rvm alias create default 1.9.3
2. 安装系统依赖包
openstack提供的ubuntu才200M+,是个很基本的系统,没有ruby环境,默认很多东西都不装的。所以要自己来。如果用的是ubuntu官方的镜像,估计就不会这么麻烦了。
系统默认安装的是ruby 1.8.7,可以直接跳至第3步。
rvm requirements
安装这些依赖包。
3. 安装bosh cli
gem install fog --no-ri --no-rdoc gem install bosh_cli gem install bosh_deployer
这样bosh cli就OK了,bosh help micro 下。
六、部署MicroBOSH
6.1 下载micro bosh stemcell
ubuntu@bosh-cli:~$ mkdir -p ~/stemcells ubuntu@bosh-cli:~$ cd stemcells ubuntu@bosh-cli:~$ bosh public stemcells --all +---------------------------------------------+-----------------------------+ | Name | Tags | +---------------------------------------------+-----------------------------+ | bosh-stemcell-0.3.0.tgz | vsphere | | bosh-stemcell-0.4.4.tgz | vsphere | | bosh-stemcell-0.4.7.tgz | vsphere | | bosh-stemcell-0.5.2.tgz | vsphere | | bosh-stemcell-aws-0.5.1.tgz | aws | | bosh-stemcell-aws-0.6.4.tgz | aws, stable | | bosh-stemcell-aws-0.6.7.tgz | aws | | bosh-stemcell-aws-0.7.0.tgz | aws, test | | bosh-stemcell-openstack-0.6.7.tgz | openstack | | bosh-stemcell-openstack-kvm-0.7.0.tgz | openstack, kvm, test | | bosh-stemcell-vsphere-0.6.1.tgz | vsphere | | bosh-stemcell-vsphere-0.6.2.tgz | vsphere | | bosh-stemcell-vsphere-0.6.3.tgz | vsphere | | bosh-stemcell-vsphere-0.6.4.tgz | vsphere, stable | | bosh-stemcell-vsphere-0.6.7.tgz | vsphere, stable | | bosh-stemcell-vsphere-0.7.0.tgz | vsphere, test | | micro-bosh-stemcell-aws-0.6.4.tgz | aws, micro, stable | | micro-bosh-stemcell-aws-0.7.0.tgz | aws, micro, test | | micro-bosh-stemcell-aws-0.8.1.tgz | aws, micro, test | | micro-bosh-stemcell-openstack-0.7.0.tgz | openstack, micro, test | | micro-bosh-stemcell-openstack-kvm-0.8.1.tgz | openstack, kvm, micro, test | | micro-bosh-stemcell-vsphere-0.6.4.tgz | vsphere, micro, stable | | micro-bosh-stemcell-vsphere-0.7.0.tgz | vsphere, micro, test | | micro-bosh-stemcell-vsphere-0.8.1.tgz | vsphere, micro, test | +---------------------------------------------+-----------------------------+ To download use `bosh download public stemcell <stemcell_name>'. For full url use --full.
bosh download public stemcell micro-bosh-stemcell-openstack-kvm-0.8.1.tgz
6.2 准备floating ip
6.3 配置部署清单
ubuntu@bosh-cli:mkdir -p deployments/microbosh-openstack ubuntu@bosh-cli:vim micro_bosh.yml
清单名称叫做micro_bosh.yml,放在 ~/deployments/microbosh-openstack目录下
name: microbosh-openstack env: bosh: password: $6$fPFEaT8CsB$sFOQhQQ/e9zZqda2a41ThKcA.AxqxiB7L9jxHUorus6cj76qnRll03b4w6DciMLOIZw4MWX23GJzld0n2L1IH1 level: DEBUG network: name: default type: dynamic label: private ip: {IPADDRESS} #floating ip resources: persistent_disk: 4096 #persistent volume cloud_properties: instance_type: test.bosh #flavor cloud: plugin: openstack properties: openstack: auth_url: http://{IPADDRESS}:5000/v2.0/tokens # openstack auth_url username: jae # openstack username api_key: jae # openstack password tenant: jae # openstack default_key_name: test default_security_groups: ["default"] private_key: ~/.ssh/test.pem
注意:
- root disk is where the os is installed, - ephemeral disk is where you can store your data, but once the instance is terminated you'll lose all data, because that disk is deleted (it's part of the image). - persistent disk is where you can store your data, and if the instance is terminated, the data is still available as the disk is not deleted, so you can create another instance, attach this persistent disk, and access the same data. So the idea behind having a persistent disk for microbosh is that all db data is stored there, and in case you recreate the microbosh (ie, update the bosh bits), you are not going to lose any data.
ubuntu@bosh-cli:~/deployments$ bosh micro deployment microbosh-openstack Deployment set to '/home/ubuntu/deployments/microbosh-openstack/micro_bosh.yml'
第二步,指定stemcell,开始部署
ubuntu@bosh-cli:~/deployments$ bosh micro deploy ~/stemcell/micro-bosh-stemcell-openstack-kvm-0.8.1.tgz Deploying new micro BOSH instance `microbosh-openstack/micro_bosh.yml' to `http://ip:25555' (type 'yes' to continue): yes Verifying stemcell... File exists and readable OK Using cached manifest... Stemcell properties OK Stemcell info ------------- Name: micro-bosh-stemcell Version: 0.8.1 Deploy Micro BOSH unpacking stemcell (00:00:12) uploading stemcell (00:02:38) creating VM from b4fa5bcc-4cd7-4f5a-9064-12d1bfaf14df (00:02:28) waiting for the agent (00:01:24) create disk (00:00:04) mount disk (00:00:15) stopping agent services (00:00:01) applying micro BOSH spec (00:00:20) starting agent services (00:00:00) waiting for the director (00:01:17) Done 11/11 00:08:47 WARNING! Your target has been changed to `http://ip:25555'! Deployment set to '/home/ubuntu/deployments/microbosh-openstack/micro_bosh.yml' Deployed `microbosh-openstack/micro_bosh.yml' to `http://ip:25555', took 00:08:47 to complete
如果顺利,10分钟内完事。主要经过了上传stemcell---->用这个stemcell制作成的镜像创建micro bosh虚拟机--->往虚拟机里植入bosh agent,用于与bosh通讯,执行bosh发的指令----> 创建volume,给db与blobstore用---->将创建的disk mount到micro bosh这个vm上---->启动agent 以及控制器director。
ubuntu@bosh-cli:~/deployments$ bosh micro deployments +---------------------+--------------------------------------+--------------------------------------+ | Name | VM name | Stemcell name | +---------------------+--------------------------------------+--------------------------------------+ | microbosh-openstack | a14b575d-1b2a-40be-8cdc-f9f444135393 | b4fa5bcc-4cd7-4f5a-9064-12d1bfaf14df | +---------------------+--------------------------------------+--------------------------------------+ Deployments total: 1
再target到micro bosh上,看看micro bosh都部署了些什么。
ubuntu@bosh-cli:~/deployments$ bosh target ip:25555 Target set to `microbosh-openstack' File gems/bosh_deployer-1.4.0/lib/bosh/cli/commands/micro.rb has been loaded as plugin but it didn't contain any commands. Make sure this plugin is updated to be compatible with BOSH CLI 1.0. Your username: admin Enter password: ***** Logged in as `admin' ubuntu@bosh-cli:~$ bosh status Updating director data... done Director Name microbosh-openstack URL http://ip:25555 Version 0.7 (release:b240cdfb bosh:57b31b3f) User admin UUID a61a1aa0-0f53-43a8-99bf-e2b30dca6174 CPI openstack dns enabled Deployment not set
micro bosh director默认端口是25555,默认用户 密码是admin admin,可以看到micro bosh 除了自身信息,还没有部署组件。部署完bosh回头看下就有了。
mkpasswd -m sha-512 yourpasswd
安装过程中可能会报mkpasswd安装不上,手工安装:
wget http://mirrors.kernel.org/ubuntu/pool/universe/w/whois/mkpasswd_5.0.0ubuntu3_amd64.deb dpkg -i mkpasswd_5.0.0ubuntu3_amd64.deb
七 遇到的问题
response => #<Excon::Response:0x00000000992b90 @body="{\"versions\": [{\"status\": \"EXPERIMENTAL\", \"id\": \"v2\", \"links\": [{\"href\": \"http://ip:9292/v2/\", \"rel\": \"self\"}]}, {\"status\": \"CURRENT\", \"id\": \"v1.1\", \"links\": [{\"href\": \"http://ip:9292/v1/\", \"rel\": \"self\"}]}, {\"status\": \"SUPPORTED\", \"id\": \"v1.0\", \"links\": [{\"href\": \"http://ip:9292/v1/\", \"rel\": \"self\"}]}]}", @headers={"Content-Type"=>"application/json", "Content-Length"=>"340", "Date"=>"Wed, 22 Aug 2012 16:38:30 GMT"}, @status=300>
期望是201,却返回了一个服务列表,什么意思。原因是openstack没有指定glance endpoint版本,有三个版本(v1.0,v1.1,v2.0),client调用的时候openstack就晕了。openstack image服务是glance。重新create glance endpoint,指定版本问题解决。如果操作详见: http://docs.openstack.org/trunk/openstack-compute/install/apt/content/osfolubuntu-identityservice.html
keystone endpoint-create \ --region RegionOne \ --service_id $GLANCE_SERVICE \ --publicurl "http://$SERVICE_HOST:9292/v1.0" \ --adminurl "http://$SERVICE_HOST:9292/v1.0" \ --internalurl "http://$SERVICE_HOST:9292/v1.0"
有说fog(iaas api的ruby封装)不支持v2.0,这个没试过,只用了1.0
Deploy Micro BOSH unpacking stemcell (00:00:10) uploading stemcell (00:02:40) creating VM from 3012d886-e75a-4acc-9040-4bd1d61f8b46 (00:02:10) Waiting for the agent |oooooo | 3/11 00:06:28 ETA: 00:11:58/usr/local/rvm/gems/ruby-1.9.3-p374/gems/excon-0.16.10/lib/excon/connection.rb:292:in `request_kernel': Expected([200, 202]) <=> Actual(400 Bad Request) (Excon::Errors::BadRequest) request => {:chunk_size=>1048576, :connect_timeout=>60, :headers=>{"User-Agent"=>"fog/1.9.0", "Content-Type"=>"application/json", "Accept"=>"application/json", "X-Auth-Token"=>"b6a85bea4ad9400bb78d09063873a053", "Host"=>"xxx:8774", "Content-Length"=>108}, :instrumentor_name=>"excon", :mock=>false, :nonblock=>true, :read_timeout=>60, :retry_limit=>4, :ssl_ca_file=>"/usr/local/rvm/gems/ruby-1.9.3-p374/gems/excon-0.16.10/data/cacert.pem", :ssl_verify_peer=>true, :write_timeout=>60, :host=>"xxx", :host_port=>"xxx:8774", :path=>"/v2/69816bacecd749f9ba1d68b3c8bae1f1/os-volumes", :port=>"8774", :query=>"ignore_awful_caching1359608037", :scheme=>"http", :body=>"{\"volume\":{\"display_name\":\"volume-a047594f-1639-408e-9738-3ac86bd67e6e\",\"display_description\":\"\",\"size\":10}}", :expects=>[200, 202], :method=>"POST"} response => #<Excon::Response:0x00000003cd8f90 @body="{\"badRequest\": {\"message\": \"The server could not comply with the request since it is either malformed or otherwise incorrect.\", \"code\": 400}}", @headers={"Content-Length"=>"141", "Content-Type"=>"application/json; charset=UTF-8", "X-Compute-Request-Id"=>"req-a96db7a9-de91-40e3-b576-788e66be63df", "Date"=>"Thu, 31 Jan 2013 04:54:06 GMT"}, @status=400> from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/excon-0.16.10/lib/excon/connection.rb:103:in `request' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/fog-1.9.0/lib/fog/core/connection.rb:21:in `request' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/fog-1.9.0/lib/fog/openstack/compute.rb:327:in `request' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/fog-1.9.0/lib/fog/openstack/requests/compute/create_volume.rb:23:in `create_volume' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/fog-1.9.0/lib/fog/openstack/models/compute/volume.rb:31:in `save' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/fog-1.9.0/lib/fog/core/collection.rb:52:in `create' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_openstack_cpi-0.0.7/lib/cloud/openstack/cloud.rb:400:in `block in create_disk' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_common-0.5.4/lib/common/thread_formatter.rb:46:in `with_thread_name' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_openstack_cpi-0.0.7/lib/cloud/openstack/cloud.rb:373:in `create_disk' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:258:in `block in create_disk' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:84:in `step' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:256:in `create_disk' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:322:in `update_persistent_disk' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:149:in `block in create' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:84:in `step' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:148:in `create' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:104:in `block in create_deployment' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:97:in `with_lifecycle' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/deployer/instance_manager.rb:103:in `create_deployment' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_deployer-1.4.0/lib/bosh/cli/commands/micro.rb:171:in `perform' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run' from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>' from /usr/local/rvm/gems/ruby-1.9.3-p374/bin/bosh:19:in `load' from /usr/local/rvm/gems/ruby-1.9.3-p374/bin/bosh:19:in `<main>' from /usr/local/rvm/gems/ruby-1.9.3-p374/bin/ruby_noexec_wrapper:14:in `eval' from /usr/local/rvm/gems/ruby-1.9.3-p374/bin/ruby_noexec_wrapper:14:in `<main>'
from /usr/local/rvm/gems/ruby-1.9.3-p374/gems/bosh_openstack_cpi-0.0.7/lib/cloud/openstack/cloud.rb:373:in `create_disk'
大概是create_disk时出错了,再看到上面几行有“create volume”,说明create_disk就是create_volume。创建volume出错。openstack的块存储服务在Folsom版中叫cinder。在openstack上调入:
cinder help
命令不存在,再ps也没有,哇,这个服务没有安装啊。大概找到问题了,安装过cinder 后问题解决。
原文链接: http://blog.csdn.net/k_james/article/details/8562666