Confd+etcd实现高可用自动发现

2017年5月10日 发表评论 阅读评论

Confd是一个轻量级的配置管理工具。通过查询Etcd,结合配置模板引擎,保持本地配置最新,同时具备定期探测机制,配置变更自动reload。其后端支持的数据类型有:etcd、consul、vault、environment variables、redis、zookeeper、dynamodb、stackengine、rancher。不过一般使用Confd和etcd的配合使用比较多。其常用架构如下:

etcd-confd-nginx

一、简单配置

1、配置etcd数据

具体步骤这里略过,这里只配置两条数据

etcdctl set /myapp/database/url www.361way.com
etcdctl set /myapp/database/user rob

2、confd安装

confd比较简单就一个文件,拿过来就可以执行,可以从github上下载:https://github.com/kelseyhightower/confd/releases ,并将其放到/usr/local/bin目录下即可。不过使用前需要创建相应的配置目录:

[root@etcd1 bin]# mkdir -p /etc/confd/{conf.d,templates}

3、创建confd配置文件

# vim  /etc/confd/conf.d/myconfig.toml
[template]
src = "myconfig.conf.tmpl"
dest = "/tmp/myconfig.conf"
keys = [
    "/myapp/database/url",
    "/myapp/database/user",
]

4、创建模板文件

# vim /etc/confd/templates/myconfig.conf.tmpl
[myconfig]
database_url = {{getv "/myapp/database/url"}}
database_user = {{getv "/myapp/database/user"}}

5、执行生成配置文件

confd -onetime -backend etcd -node http://127.0.0.1:2379    只一次
confd -interval=60 -backend etcd -node http://127.0.0.1:2379 &   按时间轮询

使用onetime参数的,配置文件生成一次后,confd程序就退出了,下面的那句,会每隔60秒轮询一次。一旦后端etcd相应的值发生变化就会重新生成相应的配置文件。

6、验证文件生成

[root@etcd1 tmp]# cat /tmp/myconfig.conf
[myconfig]
database_url = www.361way.com
database_user = rob

二、etcd+confd+nginx配置

1、创建数据

etcdctl set /myapp/subdomain myapp
etcdctl set /myapp/upstream/app2 "10.0.1.100:80"
etcdctl set /myapp/upstream/app1 "10.0.1.101:80"
etcdctl set /yourapp/subdomain yourapp
etcdctl set /yourapp/upstream/app2 "10.0.1.102:80"
etcdctl set /yourapp/upstream/app1 "10.0.1.103:80"

2、创建配置文件

# cat /etc/confd/conf.d/myapp-nginx.toml
[template]
prefix = "/myapp"
src = "nginx.tmpl"
dest = "/tmp/myapp.conf"
owner = "nginx"
mode = "0644"
keys = [
  "/subdomain",
  "/upstream",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"
# cat /etc/confd/conf.d/yourapp-nginx.toml
[template]
prefix = "/yourapp"
src = "nginx.tmpl"
dest = "/tmp/yourapp.conf"
owner = "nginx"
mode = "0644"
keys = [
  "/subdomain",
  "/upstream",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"

这里创建了两个配置文件。接下来创建一个模板文件,两个配置文件会根据该模板文件生成配置:

# cat /etc/confd/templates/nginx.tmpl
upstream {{getv "/subdomain"}} {
{{range getvs "/upstream/*"}}
    server {{.}};
{{end}}
}
server {
    server_name  {{getv "/subdomain"}}.example.com;
    location / {
        proxy_pass        http://{{getv "/subdomain"}};
        proxy_redirect    off;
        proxy_set_header  Host             $host;
        proxy_set_header  X-Real-IP        $remote_addr;
        proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
   }
}

3、验证

在进行验证启用的时候,会发现有如下报错:

[root@etcd1 conf.d]# 2017-05-08T19:06:07+08:00 etcd1 confd[10949]: INFO Target config /tmp/myapp.conf out of sync
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR "nginx: [emerg] \"upstream\" directive is not allowed here in /tmp/.myapp.conf835093196:1\nnginx: configuration file /tmp/.myapp.conf835093196 test failed\n"
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR Config check failed: exit status 1
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: INFO Target config /tmp/yourapp.conf out of sync
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR "nginx: [emerg] \"upstream\" directive is not allowed here in /tmp/.yourapp.conf196880350:1\nnginx: configuration file /tmp/.yourapp.conf196880350 test failed\n"
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR Config check failed: exit status 1

原因很简单,注意配置文件中的check_cmd 命令,该命令会进行配置文件检测,检测不通过时,配置文件不会修改,且不会执行后面的reload_cmd命令。这里想不报错也很简单,将配置文件中nginx的配置指向正确的位置,而且让nginx可以正常检测,且检测结果没有错误。再次执行命令,并修改配置文件,会发现有如下信息:

# cat myapp.conf
upstream myapp {
    server 10.0.1.100:80;
    server 10.0.1.101:80;
}
server {
    server_name  myapp.example.com;
    location / {
        proxy_pass        http://myapp;
        proxy_redirect    off;
        proxy_set_header  Host             $host;
        proxy_set_header  X-Real-IP        $remote_addr;
        proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
   }
}

三、其他

在模板文件中,会用到一些函数:map、base、exists、get、gets、getenv、datetime等 ,关于这些函数的使用,可以参看官方文档templates

etcd+confd 基本上可以和任何应用进行组合,如网上比较常见的etcd+confd+nginx 、etcd+confd+haproxy 、etcd+confd+k8s、etcd+confd+tomcat等等。和tomcat的整合,在官方上也有相应的示例,具体可以参看官方文档tomcat-sample




本站的发展离不开您的资助,金额随意,欢迎来赏!

分类: Docker/k8s/etcd 标签: ,
  1. 本文目前尚无任何评论.
  1. 本文目前尚无任何 trackbacks 和 pingbacks.